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EXECUTIVE  SUMMARY 


The  body  of  this  document  is  a  concise  description  of  suggested  best  practices, 
responsibilities,  roles,  and  procedures  for  meeting  the  Technology  Readiness  Assessment 
(TRA)  requirements  of  the  Defense  Acquisition  System.  The  intent  is  to  provide  those 
involved  with  TRAs  a  greater  understanding  of  how  TRAs  fit  into  defense  acquisition 
and  what  is  expected  by  the  DUSD(S&T),  which  serves  as  the  staff  proponent  for  TRAs 
for  the  Director  of  Defense  Research  and  Engineering  (DDR&E).  The  potential  benefit  to 
other  Office  of  the  Secretary  of  Defense  (OSD)  and  Service  Component  participants  is 
also  recognized. 

The  Department  of  Defense  (DoD)  acquisition  system  is  addressed  in  the 
following  documents: 

•  DoD  Directive  5000.1  (DoDD  5000.1),  The  Defense  Acquisition  System, 
dated  May  12,  2003 

•  DoD  Instruction  5000.2  (DoDI  5000.2),  Operation  of  the  Defense  Acquisition 
System,  dated  May  12,  2003 

•  Defense  Acquisition  Guidebook,  dated  October  2004. 

These  documents  are  available  at  http://www.akss.dau.mil/darc/darc.html.  The  Defense 
Acquisition  Guidebook  replaced  the  Interim  Defense  Acquisition  Guidebook  (October 
2002)  referenced  in  the  September  2003  version  of  this  Technology  Readiness  Assess¬ 
ment  (TRA)  Deskbook. 

A  central  theme  of  the  acquisition  process  is  that  the  technology  employed  in  sys¬ 
tem  development  should  be  “mature”  before  system  development  begins.  Normally,  for 
technology  to  be  considered  mature,  it  must  have  been  applied  in  a  prototype  article  (a 
system,  subsystem,  or  component),  tested  in  a  relevant  or  operational  environment,  and 
found  to  have  performed  adequately  for  the  intended  application.  This  implies  a  need  for 
a  way  to  measure  maturity  and  for  a  process  to  ensure  that  only  sufficiently  mature  tech¬ 
nology  is  employed.  The  Defense  Acquisition  Guidebook  provides  an  outline  of  a  process 
and  suggests  activities  for  performing  TRAs;  however,  this  guidance  is  not  mandatory. 
The  Guidebook  introduces  Technology  Readiness  Levels  (TRLs)  as  an  accepted  way  to 
describe  technology  maturity  and  suggests  activities  that  could  be  carried  out  by  Program 
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Managers  (PMs),  Component  Science  and  Technology  (S&T)  Executives,  Component 
Acquisition  Executives  (CAEs),  and  the  DUSD(S&T). 

The  body  of  this  document  includes  the  following: 

•  A  description  of  the  overall  systems  acquisition  process  in  DoD,  with  par¬ 
ticular  emphasis  on  the  roles  of  people  conducting  the  TRA  (Section  2) 

•  A  description  of  the  TRA  process  (Section  3) 

•  A  description  of  the  TRA  format  (Section  4) 

•  A  description  of  the  best  practices  for  managing  technology  maturation  (Sec¬ 
tion  5). 

The  appendixes  provide  the  following  information: 

•  Extracts  from  the  DoD  5000  Series  of  Documents  and  the  Defense  Acquisi¬ 
tion  Guidebook  relevant  to  TRAs  (Appendix  A) 

•  Extracts  from  relevant  Government  Accountability  Office  (GAO)  and  DoD 
reports  (Appendix  B) 

•  Guidance,  best  practices,  and  examples  for  assessing  technology  maturity, 
including  tables  that  provide  TRL  definitions  for  hardware,  software,  and 
manufacturing  technology  (Appendix  C) 

•  Guidance  and  best  practices  for  identifying  Critical  Technology  Elements 
(CTEs)  (Appendix  D) 

•  Policy  statements  relevant  to  the  TRA  process  (Appendix  E) 

•  A  Technology  Development  Strategy  (TDS)  template( Appendix  F) 

•  Technology  Transition  Agreement  (TTA)  elements  and  a  template  (Appen¬ 
dix  G) 

•  Specialized  definitions  and  descriptions  of  TRLs  for  biomedical  technology 
(drugs,  vaccines,  and  medical  devices)  (Appendix  H) 

•  A  discussion  of  Manufacturing  Readiness  Levels  (MRLs)  (Appendix  I). 

•  Easy-reference  displays  of  the  TRA  Activities  Time  Line  and  the  Hardware/ 
Software  TRLs  (Appendix  J).  These  11  x  17  copies  can  be  removed  from  the 
hard  copy  or  printed  from  the  soft  copy. 

The  expectation  is  that  the  basic  architecture  of  the  TRA  process  will  remain  relatively 
stable  over  time,  whereas  the  details  implementing  the  process  will  evolve  and  become 
more  or  less  explicit  over  time.  As  changes  occur,  adapting  the  appendixes  or  adding  new 
appendixes  will  provide  an  effective  way  for  the  Deskbook  to  accommodate  these 
changes. 
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SECTION  1. 
INTRODUCTION 


1.1  BACKGROUND 

The  Department  of  Defense  (DoD)  acquisition  system  is  addressed  in  the 
following  documents:1 

•  DoD  Directive  5000.1  (DoDD  5000.1),  The  Defense  Acquisition  System , 
dated  May  12,  2003 

•  DoD  Instruction  5000.2  (DoDI  5000.2),  Operation  of  the  Defense  Acquisition 
System,  dated  May  12,  2003 

•  Defense  Acquisition  Guidebook,  dated  October  2004.2 

DoDD  5000.1  and  DoDI  5000.2  provide  management  principles  and  mandatory 
policies  and  procedures  for  managing  all  acquisition  programs.  The  Guidebook  contains 
recommended  guidance  on  best  practices,  lessons  learned,  and  expectations. 

A  central  acquisition  process  theme  is  that  the  technology  should  be  “mature” 
before  system  development  begins.3  Normally,  for  technology  to  be  considered  mature,  it 
must  have  been  applied  in  a  prototype  article  (a  system,  subsystem,  or  component),  tested 
in  a  relevant  or  operational  environment,  and  found  to  have  performed  adequately  for  the 
intended  application.  This  implies  a  need  for  a  measure  of  technology  maturity  and  for  a 
process  to  ensure  that  only  sufficiently  mature  technology  is  used. 

These  needs  are  met  by  DoDI  5000.2,  which  establishes  a  requirement  for  Tech¬ 
nology  Readiness  Assessments  (TRAs),  and  by  the  Defense  Acquisition  Guidebook, 
which  suggests  a  process  and  a  methodology  for  performing  TRAs.  The  Guidebook  also 
introduces  Technology  Readiness  Levels  (TRLs)  as  an  accepted  way  to  describe 


1  These  documents  are  available  at  http://www.akss.dau.mil/darc/darc.html.  Appendix  A  contains 
relevant  extracts  from  these  documents. 

-  This  Guidebook  replaced  the  Interim  Defense  Acquisition  Guidebook  (October  2002)  referenced  in  the 
September  2003  version  of  this  Technology  Readiness  Assessment  (TRA)  Deskbook. 

3  This  reflects  a  major  conclusion  of  a  study  performed  by  the  Government  Accountability  Office 
(GAO)  (see  Appendix  B). 
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technology  maturity.4  The  National  Aeronautics  and  Space  Administration  (NASA)  has 
defined  TRLs  and  has  used  these  TRLs  in  its  program  reviews.  The  NASA  definitions  are 
the  basis  for  the  DoD  definitions. 

Regulatory  requirements  mandate  TRAs  at  the  Milestone  B  and  Milestone  C 
reviews  for  all  acquisition  programs.5  This  Deskbook  describes  actions  to  carry  out  TRAs 
that  would  normally  be  taken  by  Program  Managers  (PMs),  Component  Science  and 
Technology  (S&T)  Executives,  Component  Acquisition  Executives  (CAEs),  and  the 
Deputy  Under  Secretary  of  Defense  for  Science  and  Technology  (DUSD(S&T)).  TRAs 
for  Acquisition  Category  (ACAT)  ID6  or  ACAT  IAM7  programs  must  be  submitted  to 
the  DUSD(S&T). 

1.1.1  TRA  Definition  and  Purpose 

A  TRA  is  a  systematic,  metrics-based  process  and  accompanying  report  that 
assesses  the  maturity  of  certain  technologies  [called  Critical  Technology  Elements 
(CTEs)]8  used  in  systems.  The  report  includes  information  about  how  the  CTEs  are  iden¬ 
tified,  why  they  are  important  to  the  program,  and  an  independent  (from  the  program) 
assessment  of  their  maturity. 

The  purpose  of  the  TRA  is  to  surface  data  and  assess  information  relevant  to  the 
maturity  of  the  CTEs  in  acquisition  programs.  This  assessment  does  not  predict  future 


4  Appendix  C  addresses  the  topic  of  assessing  technology  maturity  in  some  detail. 

5  Milestone  B  initiates  system  development.  It  is  the  most  common  point  for  formal  program  initiation  in 
the  Defense  Acquisition  System.  However,  regulatory  requirements  mandate  a  TRA  for  ships  at 
program  initiation— typically  Milestone  A.  Milestone  C  approves  low  rate  initial  production  (LRIP) 
for  hardware  systems  and  initial  deployment  for  automated  information  systems  (AISs).  Section  2 
presents  an  overview  of  the  Defense  Acquisition  System. 

6  ACAT  ID  is  a  subcategory  of  ACAT  I.  ACAT  I  programs  are  Major  Defense  Acquisition  Programs 
(MDAPs)  or  programs  that  the  Milestone  Decision  Authority  (MDA)  designates  ACAT  I.  An  MDAP 
is  an  acquisition  program  that  is  not  a  highly  sensitive  classified  program  (as  determined  by  the 
Secretary  of  Defense)  but  is  designated  by  the  Under  Secretary  of  Defense  for  Acquisition,  Tech¬ 
nology,  and  Logistics  (USD(AT&L))  as  an  MDAP  based  on  several  factors,  including  research, 
development,  test,  and  evaluation  (RDT&E)  expenditures  and  procurement  expenditures.  The  MDA 
for  ACAT  ID  programs  is  the  USD(AT&L). 

7  ACAT  IAM  is  a  subcategory  of  ACAT  IA.  ACAT  IA  programs  are  Major  Automated  Information 
System  (MAIS)  programs  or  programs  designated  by  the  Assistant  Secretary  of  Defense  for  Networks 
and  Information  Integration  (ASD(NII))  [formerly  the  Assistant  Secretary  of  Defense  for  Command, 
Control,  Communications,  and  Intelligence  (ASD(C3I)]  to  be  ACAT  IA.  The  MDA  for  ACAT  IAM 
programs  is  the  ASD(NII),  who  is  also  the  Department  of  Defense  Chief  Information  Officer  (DoD 
CIO). 

8  Appendix  D  addresses  CTEs  in  some  detail. 
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performance  of  the  technologies  or  the  system  nor  does  it  assess  the  quality  of  the  system 
architecture,  design,  or  integration  plan.  It  is  simply  a  report  on  what  has  been  accom¬ 
plished  to  date  for  an  important  subset  of  technologies  in  the  program.  The  TRLs 
reported  in  the  TRA  are  part  of  the  program’s  technical  risk  assessment.  Elements  of 
technical  risk  also  include  design,  architectures,  interoperability,  cost,  schedule,  manufac¬ 
turability  and  producibility,  and  so  forth.  Thus,  although  the  PM  should  find  the  output  of 
a  disciplined  TRA  process  useful  in  highlighting  technology  items  and  shaping  the  risk 
mitigation  plans,  the  TRA  should  not  be  the  sole  means  of  discovering  technology  risk. 

To  conduct  a  TRA,  the  PM  identifies  the  CTEs  by  looking  across  the  established 
program  work  breakdown  structure  (WBS)  (or  equivalent)  for  technology  components 
that  are  essential  to  the  system  and  are  either  new  or  novel  or  are  being  applied  in  a  new 
or  novel  way.  Data  concerning  the  performance  of  the  CTEs  are  collected  and  presented 
to  reviewers  (or  teams)  who  are  independent  from  the  program  and  expert  in  the  tech¬ 
nologies.  The  independent  reviewers  assess  the  maturity  of  the  CTEs  against  established 
TRL  metrics.  This  assessment  is  approved  by  the  Service  or  Agency  S&T  Executive  and 
forwarded  to  the  CAE  or  Agency  Head,  who  then  transmits  it  to  the  DUSD(S&T).  The 
DUSD(S&T)  either  concurs  with  the  TRA,  concurs  with  reservation,  or  does  not  concur. 
If  DUSD(S&T)  does  not  concur,  it  can  either  send  the  TRA  back  to  the  Service  or 
Agency  for  changes  or  elect  to  conduct  another  Independent  Technical  Assessment 
(ITA).  In  all  cases,  the  DUSD(S&T)  forwards  recommendations  to  the  Milestone  Deci¬ 
sion  Authority  (MDA)  as  input  to  the  decision  process.9  Section  4  contains  a  suggested 
format  for  the  TRA  submitted  from  the  Service  or  Agency  S&T  component  to  the 
DUSD(S&T). 

1.1.2  The  Importance  of  the  TRA 

The  Defense  Acquisition  System  strives  to  integrate  advanced  technology  into 
producible  systems  and  deploy  these  technologies  in  the  shortest  time  practicable.  The 
TRA  and  the  recommendations  from  the  DUSD(S&T)  are  two  elements  of  the  MDA 
decision  to  transition  high-interest  programs  between  the  Defense  Acquisition  System 
milestones. 


9  For  instance,  the  DUSD(S&T)  may  recommend  remedial  action  (e.g.,  requiring  a  TRA  update  at  a  later 
time  or  using  alternative  technologies  to  replace  immature  CTEs)  to  be  included  in  the  Acquisition 
Decision  Memorandum  (ADM). 


1-3 


Money  is  saved  by  adhering  to  the  DoDI  5000.2  policy  on  technology  maturity, 
which  requires  CTEs  to  be  demonstrated  in  a  relevant  environment  before  system  devel¬ 
opment.  On  those  occasions  when  programs  have  been  initiated  with  immature  technolo¬ 
gies,  DoD  has  often  suffered  the  consequences  of  enormous  cost  growth  and  schedule 
slippages. 

Congress  has  also  voiced  its  concern  about  technology  maturity  in  acquisition 
programs.  Each  year,  the  DUSD(S&T)  is  required  to  submit  to  Congress  a  report  that 
describes  and  justifies  each  case  in  which  an  Major  Defense  Acquisition  Program 
(MDAP)  entered  system  development  with  a  CTE  that  had  not  been  demonstrated  in  a 
relevant  environment.10  The  TRAs  submitted  by  the  Services  and  Agencies  are  the  pri¬ 
mary  sources  of  data  and  information  for  that  report. 

PMs  have  found  TRAs  useful  in  trying  to  understand  the  maturity  of  their  pro¬ 
grams.  The  TRA  can  help  the  PM  by  identifying  immature  and  important  components 
and  tracking  the  maturity  development  of  those  components.  Some  programs  use  TRAs 
as  an  important  component  of  their  risk  assessment.  The  TRA  highlights  critical  tech¬ 
nologies  and  other  potential  technology  risk  areas  that  require  the  PM’s  attention. 

For  Information  Technology  (IT)  systems,  which  rely  heavily  on  off-the-shelf 
components,  TRAs  have  increased  the  focus  of  management  attention  on  CTEs  that  lie 
outside  of  hardware  and  software.  For  example,  CTEs  may  relate  to  IT  issues,  such  as 
interfaces,  throughput,  scalability,  external  dependencies,  and  information  assurance, 
depending  on  how  the  system  architecture  drives  system  interdependencies  and  complex¬ 
ities.  Since  many  IT  systems  have  experienced  problems  with  these  issues,  the  TRA  has 
proven  useful  in  understanding  potential  problems  earlier  in  the  process  when  solution 
options  are  easier  to  adopt  and  less  costly  to  implement. 

1.2  PURPOSE  OF  THIS  DOCUMENT 

This  Technology  Readiness  Assessment  (TRA)  Deskbook  gives  defense  organiza¬ 
tions  involved  with  TRAs  a  greater  understanding  of  how  TRAs  fit  into  defense  acquisi¬ 
tion  approval  process  and  what  is  expected  by  the  DUSD(S&T). 11  For  conducting  TRAs, 
it  also  contains  advice  and  best  practices  obtained  from  interviews  with  people  who  have 


10  See  Appendix  E. 

11  The  DUSD(S&T)  serves  as  the  staff  proponent  for  TRAs  for  the  Director  of  Defense  Research  and 
Engineering  (DDR&E). 
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been  involved  in  the  TRA  process.  In  addition,  this  document  provides  the  DUSD(S&T) 
staff  a  working  appreciation  of  the  overall  TRA  process,  with  enough  detail  to  allow 
them  to  meet  their  staff  responsibilities. 


1.3  ORGANIZATION  OF  THIS  DOCUMENT 

The  body  of  this  document  is  a  concise  description  of  suggested  best  practices, 
responsibilities,  roles,  and  procedures  for  meeting  the  TRA  requirements  of  the  Defense 
Acquisition  System.  It  provides 

•  A  description  of  overall  systems  acquisition  process  in  DoD,  with  particular 
emphasis  on  the  roles  of  people  conducting  the  TRA  (Section  2) 

•  A  description  of  the  TRA  process  (Section  3) 

•  A  description  of  the  TRA  format  (Section  4) 

•  A  description  of  the  best  practices  for  managing  technology  maturation 
(Section  5). 

The  10  appendixes  provide  the  following  information: 

•  Extracts  from  the  DoD  5000  Series  of  Documents  and  the  Defense  Acquisi¬ 
tion  Guidebook  relevant  to  TRAs  (Appendix  A) 

•  Extracts  from  relevant  Government  Accountability  Office  (GAO)  and  DoD 
reports  (Appendix  B) 

•  Guidance,  best  practices,  and  examples  for  assessing  technology  maturity, 
including  tables  that  provide  TRL  definitions  for  hardware,  software,  and 
manufacturing  technology  (Appendix  C) 

•  Guidance  and  best  practices  for  identifying  Critical  Technology  Elements 
(CTEs)  (Appendix  D) 

•  Policy  statements  relevant  to  the  TRA  process  (Appendix  E) 

•  A  Technology  Development  Strategy  (TDS)  template( Appendix  F) 

•  Technology  Transition  Agreement  (TTA)  elements  and  a  template  (Appen¬ 
dix  G) 

•  Specialized  definitions  and  descriptions  of  TRLs  for  drugs,  vaccines,  and 
medical  devices  (Appendix  H) 

•  A  discussion  of  Manufacturing  Readiness  Levels  (MRLs)  (Appendix  I). 

•  Easy-reference  displays  of  the  TRA  Activities  Time  Line  and  the  Hardware/ 
Software  TRLs  (Appendix  J).  These  11  x  17  copies  can  be  removed  from  the 
hard  copy  or  printed  from  a  soft  copy. 
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1.4  CHANGES  FROM  THE  PREVIOUS  VERSION 


This  document  replaces  the  Technology  Readiness  Assessment  (TRA)  Deskbook 
published  September  2003.  This  current  version  provides  a  more  up-to-date  discussion  of 
the  process  and  best  practices  for  performing  a  TRA.  It  also  provides  much  greater  detail 
in  identifying  and  assessing  CTEs  and  the  readiness  of  critical  manufacturing  technolo¬ 
gies,  conducting  TRAs  for  Major  Automated  Information  System  (MAIS)  programs  and 
the  IT  aspects  of  MDAPs,  and  defining  software  TRLs. 
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SECTION  2. 

SYSTEMS  ACQUISITION  OVERVIEW 


2.1  SYSTEMS  ACQUISITION  FRAMEWORK 

Figure  2-1  depicts  the  defense  acquisition  process.  TRAs  are  conducted  just 
before  Milestones  B  and  C  and  at  program  initiation  (typically  Milestone  A)  for  ships. 
The  TRAs  are  a  component  of  the  Milestone  decisions. 


Process  entry  at  Milestones  A,  B,  or  C 
Entrance  criteria  met  before  entering  phase 

Evolutionary  Acquisition  or  Single  Step  to  Full 
Capability 


A  A! 


(Program 

Initiation) 


IOC 


FOC 


Concept 

Refinement 

Technology 

Development 

System  Development 
&  Demonstration 

Production  & 
Deployment 

Operations  & 
Support 

\  Concept 
/  Decision 

/\  Design 
\  >  Readiness 

V  Review 

LRIP/IOT&E  Decision 

V  Review 

Pre-Systems  Acquisition 


Systems  Acquisition 


Sustainment 


Figure  2-1.  Defense  Acquisition  Management  Framework 
(Source:  DoDI  5000.2) 

The  acquisition  process  is  progressive,  proceeding  from  desired  operational  capa¬ 
bilities  to  system  concepts  and  then  successively  to  systems,  subsystems,  and  components 
that  are  more  sharply  and  more  completely  defined.  The  following  description  of  the 
acquisition  system  focuses  on  the  elements  that  impact  technology  selection,  develop¬ 
ment,  and  use  in  defense  system  acquisition.  DoDI  5000.2  and  the  Defense  Acquisition 
Guidebook  contain  a  more  complete  description  of  the  acquisition  system. 


2.2  PRE-CONCEPT  DECISION 

The  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  identifies 
current  and  future  gaps  in  our  ability  to  carry  out  Joint  warfighting  missions  and 
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functions.12  The  identification  process  begins  before  the  Concept  Decision  point  shown 
in  Figure  2- 1  at  the  start  of  the  acquisition  management  framework. 

Analysis  conducted  as  a  part  of  the  JCIDS  process  results  in  doctrine,  organiza¬ 
tion,  training,  materiel,  leadership  and  education,  personnel,  and  facilities  (DOTMLPF) 
change  recommendations  and/or  an  Initial  Capabilities  Document  (ICD),  along  with  an 
associated  Analysis  of  Alternatives  (AoA)  plan.  The  ICD,  as  approved  by  the  Joint 
Requirements  Oversight  Council  (JROC),  validates  the  capability  need  and  includes  a  list 
of  alternative  materiel  approaches.  These  approaches  establish  boundary  conditions  for 
the  subsequent  AoA,  which  analytically  compares  these  material  approaches.  Materiel 
approaches  are  elements  of  the  system  concepts  discussed  in  DoDI  5000.2. 13 

Before  planning  the  AoA,  robust  analyses  of  the  technology14  needed  by  each  of 
the  candidate  concepts  must  be  conducted  to  provide  assurance  that  the  technology  is 
available  or  can  be  developed.  Therefore,  before  concept  decision,  CTE  identification 
should  be  a  major  effort.  Studies  must  be  carried  out  in  enough  detail  to  reveal  those  fea¬ 
tures  that  will  be  difficult  or  impossible  to  achieve  with  currently  mature  technology. 
This  includes  a  thorough  consideration  of  alternative  technologies,  with  focus  on  their 
maturity,  risk,  and  maturation  needs. 

2.3  CONCEPT  DECISION  AND  THE  CONCEPT  REFINEMENT  PHASE 

The  ICD  and  a  plan  for  an  AoA  are  presented  to  the  MDA  at  a  Concept  Decision 
review.  The  MDA  designates  the  lead  DoD  component(s),  grants  approval  to  proceed 
into  Concept  Refinement,  approves  a  selected  concept,  approves  the  AoA  plan,  and 
establishes  a  date  for  a  Milestone  A  review. 

Paragraph  3.5.3  of  DoDI  5000.2  states  that  the  “AoA  shall  assess  the  critical  tech¬ 
nologies  associated  with  these  concepts,  including  technology  maturity,  technical  risk, 
and,  if  necessary,  technology  maturation  and  demonstration  needs.”  Therefore,  the 
system  concept  should  become  more  definitive  as  the  AoA  progresses.  This  narrowing  of 


12  This  discussion  has  been  extracted  from  the  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI) 
3170.01E,  Joint  Capabilities  Integration  and  Development  System,  11  May  2005,  and  the  Chairman  of 
the  Joint  Chiefs  of  Staff  Manual  (CJCSM)  3170.01B,  Operation  of  the  Joint  Capabilities  Integration 
and  Development  System,  11  May  2005. 

13  The  term  “concepts,”  as  used  in  DoDI  5000.2,  refers  to  more  than  materiel  concepts. 

14  DoDI  5000.2,  paragraph  3.4.1  states  that  the  JCIDS  process  “shall  include  robust  analyses  that 
consider  affordability,  technology  maturity,  and  responsiveness.” 
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choices  should  be  based  on  the  availability  of  mature  technology  and  many  other 
considerations.  It  is  an  essential  ingredient  of  the  Technology  Development  Strategy 
(TDS),  which  is  a  plan  for  maturing  the  promising  but  still  unproven  technologies  that  are 
key  to  the  system  concept.  The  TDS  is  a  requirement  for  Milestone  A.  It  is  the  basis  not 
only  for  the  Technology  Development  phase  of  the  acquisition  process,  but  also  for  the 
program’s  acquisition  strategy  at  Milestone  B.15 

2.4  MILESTONE  A  AND  THE  TECHNOLOGY  DEVELOPMENT  PHASE 

At  Milestone  A,  the  program  enters  the  Technology  Development  phase  and 
begins  execution  of  the  MDA-approved  TDS.  Note:  At  this  point  in  the  process,  the 
program  is  still  not  considered  an  acquisition  program.16 

As  stated  in  DoDI  5000.2,  paragraph  3.6.1,  “the  purpose  of  this  phase  [the  Tech¬ 
nology  Development  phase]  is  to  reduce  technology  risk  and  to  determine  the  appropriate 
set  of  technologies  to  be  integrated  into  a  full  system.”  In  DoD,  “sufficiently  mature  for 
product  development”  normally  requires  the  technologies  to  be  demonstrated  in  a  rele¬ 
vant  environment,  17  so  that  after  detailed  design,  these  items  should  be  suitable  for  inte¬ 
gration  into  the  full  system. 

Technology  Development  is  often  executed  with  a  Technology  Transition  Agree¬ 
ment  (TTA).  This  document  is  a  commitment  of  the  requirements/resource  sponsor,  S&T 
activity  (developer  and  provider  of  the  technology/product),  and  acquisition  program 
sponsor  (intended  receiver  of  a  technology  or  capability)  to  develop,  deliver,  and  inte¬ 
grate  a  technology /product  into  an  acquisition  program.18 

Near  the  completion  of  the  Technology  Development  phase,  a  TRA  is  conducted. 
The  purpose  of  this  TRA  is  to  ensure  that  a  program  does  not  enter  System  Development 
and  Demonstration  (SDD)  relying  on  technologies  that  fail  to  meet  the  maturity  criterion. 
This  means  that  technologies  important  to  the  system  development  and  production  have 
been  identified  through  a  credible  process  and  that  the  maturities  of  these  technologies 
have  been  demonstrated  at  an  appropriate  level.  The  Milestone  B  TRA  includes  primarily 


15  The  TDS  describes  how  the  program  will  be  divided  into  technology  spirals  and  development  incre¬ 
ments,  what  prototypes  will  be  built  and  tested,  and  specific  exit  criteria.  Appendix  F  contains  a  TDS 
template  used  by  the  Defense  Acquisition  University  (DAU). 

1 6  Shipbuilding  acquisition  programs  can  be  initiated  at  Milestone  A  (see  DoDI  5000.2,  paragraph  3.6.3). 

17  The  relevant  environment  is  described  in  detail  in  Appendix  D. 

18  See  Appendix  G  for  a  TTA  template. 
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technical  information  about  the  system,  such  as  a  WBS,  and  a  system  architecture  with 
CTEs  and  results  from  tests  that  included  prototypes  or  models. 

2.5  MILESTONE  B  AND  THE  SDD  PHASE 

A  favorable  Milestone  B  decision  authorizes  a  program  or  an  increment  of  a  pro¬ 
gram  to  enter  SDD.19  The  SDD  phase  consists  of  two  major  efforts  (System  Integration 
and  System  Demonstration)  and  a  mid-phase  Design  Readiness  Review  (DRR)  per  Fig¬ 
ure  2-1.  During  System  Integration,  the  chosen  technologies  and  subsystems  are  inte¬ 
grated  into  a  detailed  system  design.  This  effort  typically  includes  the  demonstration  of 
prototype  articles  or  Engineering  Development  Models  (EDMs)  that  result  from  the 
integration  of  some  or  all  of  the  subsystems.  The  DRR  marks  the  transition  to  System 
Demonstration.  During  System  Demonstration,  system-level  prototypes  are  demonstrated 
in  the  intended  environment  to  show  that  the  system  can  meet  approved  requirements.20 
This  effort  must  also  establish  that  no  significant  manufacturing  risks  exist  and  the 
needed  industrial  capabilities  will  be  available. 

A  new  or  revised  TRA  is  required  before  Milestone  C.  This  TRA  should 

•  Reflect  the  resolution  of  any  technology  deficiencies  that  arose  during  SDD 

•  Establish  that  all  critical  manufacturing  technologies  are  mature  for  hardware 
systems 

•  Document  successful  development,  test,  and  evaluation  (DT&E)  for  MAIS 
acquisitions  and  software-intensive  systems. 

2.6  MILESTONE  C:  ENTRY  TO  PRODUCTION  AND  DEPLOYMENT 

Milestone  C  authorizes  Low  Rate  Initial  Production  (LRIP)  for  MDAPs  or  limited 
deployment  in  support  of  operational  testing  for  MAIS  programs  or  software-intensive 
systems  that  have  no  production  components.  LRIP  produces  production-representative 


19  The  Joint  Staff  finalizes  a  Capability  Development  Document  (CDD)  that  is  validated  and  approved 
before  Milestone  B.  The  CDD  captures  the  information  necessary  to  develop  a  proposed  program.  The 
CDD  outlines  an  affordable  increment  of  militarily  useful,  logistically  supportable,  and  technically 
mature  capability. 

20  After  DRR,  the  Joint  Staff  finalizes  a  Capability  Production  Document  (CPD).  The  CPD  is  validated 
and  approved  before  Milestone  C.  Key  performance  parameters  (KPPs)  from  the  CPD  are  inserted 
verbatim  into  the  acquisition  strategy  and  the  Acquisition  Program  Baseline  (APB).  See  CJCSM 
3170.01B,  Operation  of  the  Joint  Capabilities  Integration  and  Development  System,  dated  11  May 
2005,  Enclosure  G,  paragraphs  1  and  2.  This  manual  should  be  available  at  http://www.dtic.mil/ 
cjcs_directives/cjcs/manuals.htm. 
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articles  for  Initial  Operational  Test  and  Evaluation  (IOT&E)  and  completes  manufac¬ 
turing  development  to  ensure  efficient  manufacturing  capability.  After  LRIP,  approval  for 
Full  Rate  Production  (FRP)  depends  on  demonstrating  that  critical  manufacturing  pro¬ 
cesses  are  under  control  in  a  production  environment,  statistical  process  control  data  are 
being  collected,  and  design-to-cost  (DTC)  goals  have  been  met. 21 

2.7  TAILORING  THE  ACQUISITION  PROCESS 

The  acquisition  process  framework  can  be  tailored  to  a  specific  acquisition  pro¬ 
gram  structure.  For  example,  the  program  does  not  have  to  start  at  Concept  Refinement. 
It  can  start  at  any  point  consistent  with  phase-specific  entrance  criteria  and  statutory 
requirements.  However,  any  program  that  enters  the  acquisition  process  mid-phase  must 
meet  the  entrance  requirements  of  that  phase.  In  practice,  this  means  that  programs 
starting  at  or  beyond  Milestone  B  must  conduct  an  associated  TRA  to  ensure  that  the 
technology  is  ready  for  the  upcoming  phase  of  acquisition. 

DoDI  5000.2  establishes  evolutionary  development  as  the  strategy  that  DoD 

prefers: 


3.3.2.  The  approaches  to  achieve  evolutionary  acquisition  require  col¬ 
laboration  between  the  user,  tester,  and  developer.  They  include 

3.3.2.1.  Spiral  Development.  In  this  process,  a  desired  capability  is  iden¬ 
tified,  but  the  end- state  requirements  are  not  known  at  program  initiation. 

Those  requirements  are  refined  through  demonstration  and  risk  manage¬ 
ment;  there  is  continuous  user  feedback;  and  each  increment  provides  the 
user  the  best  possible  capability.  The  requirements  for  future  increments 
depend  on  feedback  from  users  and  technology  maturation. 

3.3.2.2.  Incremental  Development.  In  this  process,  a  desired  capability  is 
identified,  an  end-state  requirement  is  known,  and  that  requirement  is  met 
over  time  by  developing  several  increments,  each  dependent  on  available 
mature  technology. 

To  ensure  that  the  technology  is  mature,  a  TRA  should  be  conducted  for  each 
increment  or  spiral  under  either  approach  to  evolutionary  acquisition.22 


21  From  DoDI  5000.2,  paragraph  3. 8. 3.4:  “Software  shall  have  demonstrated  the  maturity  level  required 
in  the  CPD  prior  to  deploying  it  to  the  operational  environment.” 

22  DoDI  5000.2,  paragraph  3. 7. 2.4  and  Table  E3.T2. 
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SECTION  3. 
THE  TRA  PROCESS 


3.1  INTRODUCTION 

This  section  describes  the  overall  TRA  process.  While  the  focus  is  on  the  general 
procedures  and  best  practices  for  ACAT  ID  and  IAM  programs,  much  of  the  material  is 
also  applicable  to  smaller  programs.  Figure  3-1  portrays  a  suggested  time  line  for  TRA 
activities.  Subsections  3.2  and  3.3  discuss  activities,  roles,  and  responsibilities  of  key 
players  within  the  two  broad  functional  areas  that  make  up  the  overall  TRA  process. 
Examples  of  some  best  practices  for  each  of  these  functional  areas  are  as  follows: 

•  Identifying  CTEs 

-  The  PM  is  responsible  for  identifying  CTEs.  Technologies  may  be  criti¬ 
cal  from  a  performance  perspective,  a  manufacturing  process,  or  from  a 
material,  measurement,  or  tooling/infrastructure  perspective. 

•  Assessing  CTE  Readiness 

-  The  Component  S&T  Executive  should  direct  the  TRA  and  decide  how 
it  will  be  conducted.  Typically,  much  of  the  information  used  in  a  TRA 
comes  from  the  PM;  however,  the  assessment  should  be  independent  of 
the  PM.  TRLs  provide  the  metric  for  a  technology’s  maturity. 

This  Deskbook  provides  guidance  on  best  practices  observed  across  the  board. 
Each  Component  has  its  own  procedures  for  conducting  TRAs.  Refer  to  the  Service  or 
Agency  S&T  Component  for  details. 

3.2  IDENTIFYING  CTEs23 

Figure  3-2  shows  a  representative  schedule  of  activities  to  identify  CTEs  for  a 
TRA.  The  “weeks”  shown  across  the  top  of  the  figure  represent  the  number  of  weeks 
before  a  milestone  decision.  Depending  on  the  size  and  complexity  of  the  program,  the 
start  point  and  activity  length  may  vary  greatly.  ACAT  ID  and  IAM  programs  should 


23  See  Appendix  D  for  more  details. 
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Task  Name 


m  mi  ms  ms  m  m  m  m  ma  M9  mb  mi  mb  ms  M4  mb  m2  mi  mo  m  m  m  m  m  m  m  m 


PM  Notifies  CS&T  Exec  and  DUSOfS^T)  of  the  date  for  MS  review  meeting 
DUSD(S&T)  Appoints  Action  Officer  (AO)  and  so  Notifies  PM  and  CS&T  Exec 
PM,  CSSJ  Exec,  and  DUSD(S&T)/AO  Agree  on  TRA  Schedule 
CTE  Identification  Process* 

CTE  TRL  Evaluation  Data  Collection* 

PM  Identifies  CTEs  to  CSSJ  Exec  and  DUSD(SSJ) 

Plvl  and  CS&.T  Exec  Agree  on  CTEs 


CS&T  Exec  Directs  TRA  (Copy  to  AO) 


Component  TRA  is  Performed 

CS&T  Exec  Sends  TRA  to  Component  Acquisition  Exec  (CAE)  and  Copy  to  DUSD(S&T) 
CAE  Accepts  TRA  Findings  or  Reconciles  Them  with  the  PM 
AO  informs  CS&T  Exec  of  adequacy  of  TRA  and  organizes  evaluation  of  TRA 
CAE  Sends  Endorsed  TRA  Findings  to  DUSD(S&T),  with  Notation  of  any  Changes 


AO  Leads  DUSD(SSI)  Evaluation  of  TRA 


AO  Briefs  DUSD(S&T)  on  Evaluation  Status 
(If  necessary,  Independent  Technical  Asessment  (ITA)  Directed  and  Conducted} 
DUSD(S&T)  Sends  Results  of  Evaluation  or  ITA  to  DIPT  and  DAB  or  ITAB 
Milestone  Review  Meeting 


These  actions  can  occur  as  much  as 
3  years  before  the  milestone. _ 


Figure  3-1 .  Suggested  Time  Line  for  TRA  Activities  for  ACAT  ID  and  1AM  Programs 
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TRA  Schedule  Established 
CTE  Identification  Process 
Data  Collection 
CTEs  Coordinated 


Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week 
26  25  24  23  22  21  20  19  18  17  16  15 

A 


Figure  3-2.  Representative  Schedule  for  Identifying  CTEs 

begin  the  TRA  process  a  minimum  of  26  weeks  (preferably  52  weeks)  before  the  mile¬ 
stone  decision.  The  following  subsections  describe  the  activities  for  each  line  in  Fig¬ 
ure  3-2.  These  descriptions  include  key  player  roles  and  responsibilities  and  the  most 
important  best  practices. 


3.2.1  TRA  Schedule  Established 

Six  to  12  months24  before  a  Milestone  B  or  C  review  (or  program  initiation  in  the 
case  of  ships),  the  TRA  process  begins  when  the  Component  S&T  Executive,  working 
closely  with  the  PM,  establishes  a  schedule  for  conducting  the  TRA.  The  schedule  should 
be  consistent  with  the  program’s  integrated  master  schedule.  The  TRA  should  be  com¬ 
pleted  at  least  6  weeks  before  the  Milestone  review  to  allow  sufficient  time  for  the 
DUSD(S&T)  to  conduct  its  review  and,  if  needed,  to  request  TRA  revisions  and/or  an 
ITA. 

Key  Player  Roles  and  Responsibilities  in  Establishing  the  TRA  Schedule 

•  Component  S&T  Executive.25  Develop 
the  TRA  schedule  jointly  with  the  program 
office.  The  schedule  should  be  coordinated 
with  the  DUSD(S&T)  for  ACAT  ID  and 
ACAT  IAM  acquisitions.  This  not  only 
allows  the  Office  of  the  Secretary  of 
Defense  (OSD)  adequate  time  to  prepare, 
but  also  provides  an  opportunity  for  OSD  to  share  information  on  high-inter¬ 
est  items.  Provide  training  and  support  to  the  program  office  concerning  its 
roles  and  responsibilities  in  the  TRA  process. 

•  Agency  Head.  When  a  program  is  not  managed  by  one  of  the  Components, 
the  head  of  the  lead  Agency  should  designate  a  person  (e.g.,  the  CIO)  to 


Best  Practice 
Coordinate  the  TRA 
schedule  with  the 
bUSb(S&T). 


24  The  time  varies  as  a  function  of  Component  procedures  and  the  complexity  of  the  program. 

25  The  Component  S&T  Executive  can  delegate  his  (or  her)  roles  and  responsibilities  to  a  TRA  coordina¬ 
tor  elsewhere  in  the  organization. 
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carry  out  the  Component  S&T  Executive’s  TRA  roles  and  responsibilities  if 
that  position  does  not  exist  in  the  Agency.  The  person  selected  should  be 
competent  in  the  technical  area  of  the  program,  independent  of  the  program, 
and  knowledgeable  about  the  DoD  acquisition  process. 

•  PM.  Support  the  Component  S&T  Executive  in  developing  and  coordinating 
the  schedule.  Designating  a  responsible  individual  in  the  program  office  to 
organize  all  TRA  activities  is  helpful.  That  individual  should  be  the  interface 
point  between  the  Component  S&T  Executive  and  the  DUSD(S&T). 

3.2.2  The  CTE  Identification  Process 

The  working  definition  for  a  CTE  is  as  follows: 

A  technology  element  is  “critical”  if  the  system  being  acquired  depends  on 
this  technology  element  to  meet  operational  requirements  (with  accept¬ 
able  development,  cost,  and  schedule  and  with  acceptable  production 
and  operation  costs)  and  if  the  technology  element  or  its  application  is 
either  new  or  novel.  Said  another  way,  an  element  that  is  new  or  novel  or 
is  being  used  in  a  new  or  novel  way  is  critical  if  it  is  necessary  to  achieve 
the  successful  development  of  a  system,  its  acquisition,  or  its  operational 
utility. 

CTE  identification  is  fundamental  to  the  TRA  concept.  For  a  readiness  assessment 
to  be  useful,  it  must  include  all  CTEs.  CTEs  should  be  identified  in  the  context  of  the 
program’s  systems  engineering  process  based  on  a  comprehensive  review  of  the  pro¬ 
gram’s  established  WBS.  For  IT/MAIS  systems,  the  system  architecture  should  be  used. 

In  the  CTE  identification  process  line  of  Figure  3-2,  the  dashed  line  to  the  left  of 
“Week  26”  indicates  that  much  of  this  CTE  identification  should  occur  well  before  the 
formal  TRA  process.  In  fact,  most  CTEs  should  be  identified  during  Concept  Refine¬ 
ment.  The  TDS  should  reflect  the  result  of  a  process  sufficiently  thorough  and  disciplined 
to  identify  those  technologies  (including  CTEs)  that  have  a  realistic  potential  to  be 
exploited  beneficially  in  the  Technology  Development  phase.  Failure  to  recognize  the 
CTEs  at  this  stage  will  usually  result  in  wasting  resources— time,  money,  facilities,  and 
so  forth— and  could  result  in  an  unfavorable  Milestone  B  decision.  As  system  devel¬ 
opment  proceeds,  the  possibility  exists,  through  necessity  or  opportunity,  for  the  exploi¬ 
tation  of  technologies  not  previously  considered.  These  technologies  must  be  given 
careful  consideration  to  decide  whether  they  are  critical  and  whether  they  are  mature 
enough  to  be  included  in  the  detailed  design. 
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It  may  require  10  weeks  or  more  to  finalize  the  list  of  CTEs  for  the  TRA  because 
CTE  identification  takes  place  in  two  stages.26  The  PM  should  prepare  an  initial  list  of 
possible  CTE  candidates.  An  independent  panel  should  be  used  to  determine  which  of  the 
candidate  technologies  included  in  the  original  list  are  “truly”  critical  based  on  the  CTE 
definition  (see  shaded  box  on  page  3-6). 

Several  useful  questions  have  been  developed  to  facilitate  this  process: 

1.  Does  the  technology  directly  impact  an  operational  requirement? 

2.  Does  the  technology  have  a  significant  impact  on  an  improved  delivery 
schedule? 

3.  Does  the  technology  have  a  significant  impact  on  the  affordability  of  the 
system? 

4.  If  this  is  a  spiral  development,  is  the  technology  essential  to  meet  the  spiral 
deliverables? 

5.  Is  the  technology  new  or  novel? 

6.  Has  the  technology  been  modified? 

7.  Has  the  technology  been  repackaged  such  that  a  new  relevant  environment  is 
realized? 

8.  Is  the  technology  expected  to  operate  in  an  environment  and/or  achieve  a  per¬ 
formance  beyond  its  original  design  intention  or  demonstrated  capability? 

For  a  technology  to  be  critical,  the  answer  to  one  of  the  first  four  questions  (1-4)  must  be 
“yes,”  and  the  answer  to  one  of  last  four  questions  (5-8)  must  also  be  “yes.”  In  ques¬ 
tions  5  to  8,  the  environments  could  include  the  following: 

•  Physical  Environment.  For  instance,  mechanical  components,  processors, 
servers,  and  electronics;  kinetic  and  kinematic;  thermal  and  heat  transfer; 
electrical  and  electromagnetic;  climatic— weather,  temperature,  particulate; 
network  infrastructure 

•  Logical  Environment.  For  instance,  software  (algorithm)  interfaces;  secu¬ 
rity  interfaces;  Web-enablement 

•  Data  Environment.  For  instance,  data  formats  and  databases;  anticipated 
data  rates,  data  delay  and  data  throughput;  data  packaging  and  framing 


26  Two  stages:  preparing  a  list  of  potential  CTEs  and  conducting  a  review  to  determine  which  CTEs  are 
really  critical. 
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•  Security  Environment.  For  instance,  connection  to  firewalls;  security  appli¬ 
ques;  rates  and  methods  of  attack 

•  User  and  Use  Environment.  For  instance,  scalability;  upgradability;  user 
behavior  adjustments;  user  interfaces;  organizational  change/realignments 
with  system  impacts;  implementation  plan. 

Therefore,  additional  questions  that  can  help  guide  the  definition  of  environment 
for  the  CTE  candidates  include  the  following: 

•  Is  the  physical/logical/data  environment  in  which  this  CTE  has  been  demon¬ 
strated  similar  to  the  intended  environment?  If  not,  how  is  it  different?  Is  the 
difference  important? 

•  Is  the  CTE  going  to  be  operating  at  or  outside  of  the  usual  performance  enve¬ 
lope?  Do  the  design  specifications  address  the  behavior  of  the  CTE  under 
these  conditions?  What  is  unique  or  different  about  this  proposed  operations 
environment? 

•  Do  test  data,  reports,  or  analyses  that  compare  the  demonstrated  environment 
to  the  intended  environment  exist?  If  modeling  and  simulation  (M&S)  is  an 
important  aspect  of  that  comparison,  are  the  analysis  techniques  common  and 
generally  accepted? 

CTEs  may  also  include  high-leverage  and/or  high-impact  manufacturing  tech¬ 
nologies  that  enable  faster  delivery  of  affordable  systems  if  there  is  something  not  well 
characterized  or  understood  about  them  or  their  use  for  producing  the  system.  For  a 
manufacturing  technology,  the  following  questions  replace  previous  questions  5  to  8.  The 
answer  to  a  least  one  of  them  must  be  “no”  for  the  technology  to  be  considered  a  CTE. 

1.  Has  the  manufacturing  technology  been  successfully  integrated  into  a  pro¬ 
duct  line? 

2.  Is  the  industrial  base27  capable  of  design,  development,  production,  mainte¬ 
nance  and  support,  and  disposal  of  the  system? 

3.  Is  the  intended  design  producible? 

4.  Have  the  materials  been  characterized  in  a  manufacturing  environment? 

5.  Are  the  materials  available  to  meet  quantity  and  schedule  demands? 

6.  Are  the  DTC  goals  achievable? 

7.  Are  the  key  manufacturing  processes  characterized,  capable,  and  controllable 
with  respect  to  achieving  the  system  requirements? 


27  Depending  on  the  circumstances,  this  may  be  limited  to  the  National  industrial  base. 
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Key  Player  Roles  and  Responsibilities  in  the  CTE  Identification  Process 

•  PM.  Within  the  context 
of  the  program’s  systems 
engineering  approach  and 
based  on  a  compre¬ 
hensive  review  of  the 
program’s  established 
WBS  or  system  architecture,  use  the  CTE  definition  to  prepare  an  initial  list 
of  possible  CTE  candidates.  When  competing  designs  exist,  identify  CTEs 
separately  for  each  one.  At  Milestone  C,  begin  with  CTEs  identified  at  Mile¬ 
stone  B.  However,  unplanned  performance  and  especially  manufacturing 
technologies  could  have  been  incorporated  in  the  design  during  SDD,  so  a 
careful  review  should  be  conducted  at  Milestone  C  to  find  any  newly 
emergent  CTEs. 

Some  programs  rely  heavily 
on  commercial-off-the-shelf 
(COTS)  or  government-off- 
the-shelf  (GOTS)  compo¬ 
nents  and  may  have  little-to- 
no  technology  development 
before  Milestone  B.  Also, 
these  programs  often  plan  for  a  competitive  acquisition  strategy,  and,  conse¬ 
quently,  the  design  approach(es)  and  the  associated  CTEs  will  not  be  final¬ 
ized  until  the  SDD  contract  is  awarded.  Such  a  situation  is  contrary  to  good 
systems  engineering.  According  to  the  Defense  Acquisition  Guidebook,  the 
Systems  Requirements  Review  (SRR)  conducted  before  Milestone  B  is  a 
multifunctional  technical  review.  The  purpose  of  this  review  is  to  ensure  that 
all  system  and  performance  requirements  are  defined  and  consistent  with  cost 
(program  budget),  time  frame  (program  schedule),  risk,  and  other  system 
constraints.  Among  other  things,  the  SRR  is  intended  to  ensure  consistency 
between  the  system  requirements  and  the  preferred  system  solution  and 
available  technologies.  It  includes  a  preliminary  allocation  of  system 
requirements  to  hardware,  human,  and  software  subsystems  and  an  identifi¬ 
cation  of  all  software  components  (tactical,  support,  deliverable,  nondeliver¬ 
able,  and  so  forth). 

If  some  overriding  circumstance 
dictates  that  the  program  has  a  Mile¬ 
stone  B  review  before  SDD  propos¬ 
als  are  received,  discuss  the  options 
with  the  DUSD(S&T)  as  early  as 


Best  Practice 

When  the  CTEs  are  uncer¬ 
tain,  discuss  options  with 
the  DUSD(S<&T)  as  early  as 
possible. 


Best  Practice 

To  the  extent  possible,  exploit 
the  SRR  before  Milestone  B  as  a 
contributor  to  the  CTE  identifi¬ 
cation  process. 


Best  Practice 

Apply  the  CTE  definition  across  the 
system  WBS  or  system  architecture 
to  identify  CTE  candidates. 
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possible.  If  a  decision  on  contract  award  is  made  before  Milestone  B,  use  the 
winning  proposal  as  the  basis  for  a  TRA.  If  proposals  have  been  received  but 
the  contract  award  decision  will  occur  during  or  after  the  Milestone  B  review, 
identify  and  assess  any  CTE  bid  in  any  of  the  proposals.28  In  either  case,  the 
TRA  may  be  proprietary  and/or  competition  sensitive. 

If  a  program  integrates  critical  systems  or  subsystems  being  developed  in 
other  programs,  the  PM  of  the  higher  order  program  (in  preparation  for  a 
TRA)  should  identify  the  CTEs— including  interface  technologies  — used  on 
his/her  side  of  the  interfaces.  This  PM  should  request  [through  the  appropri¬ 
ate  Program  Executive  Office  (PEO)  or  CAE,  as  necessary]  and  obtain  the 
identification  of  any  CTEs  in  the  lower  order  programs.  The  CTEs  of  the 
higher  order  system  and  all  lower  order  systems  or  subsystems  should  be 
included  in  the  list  of  CTEs  that  the  PM  of  the  higher  order  system  submits  to 
the  Component  S&T  Executive  and  the  DUSD(S&T). 

•  Component  S&T  Execu¬ 
tive.  In  conjunction  with 
the  PM,  form  an  inde¬ 
pendent  panel  to  review 
the  candidate  technologies 
included  in  the  original 
PM-generated  list  and, 
based  on  the  CTE  definition,  make  recommendations  on  which  of  these  tech¬ 
nologies  are  “truly”  critical  elements. 

An  Action  Officer  (AO),  appointed  by  the  Component  S&T  Executive, 
should  participate  in  the  identification  process  to  the  extent  that  his/her  par¬ 
ticipation  is  considered  useful  and  valuable.  The  AO  can  provide  beneficial 
TRA  process  and  policy  experience  and  information  and  can  also  minimize 
the  chance  that  an  unexpected  problem  will  delay  the  process.  The  AO 
should  understand  the  reasons  for  the  inclusion  and  exclusion  of  technologies 
from  the  initial  candidate  list  before  the  list  is  shown  to  the  DUSD(S&T). 

•  Independent  Panel.  On  the  basis  of  the  CTE  definition,  the  PM’s  answers  to 
questions,  and  personal  experience  of  panel  members,  make  final 
recommendations  (with  associated  rationale)  on  which  CTEs  should  be 
assessed  in  the  TRA. 


Best  Practice 

Obtain  advice  from  an  independent 
expert  panel  on  which  CTE  candi¬ 
date  technologies  should  be  included 
on  the  final  CTE  list. 


Another  circumstance  in  which  CTEs  may  not  be  firmly  understood  is  the  program  initiation  TRA  for 
ships.  Similar  best  practices  apply.  If  decisions  on  technology  development  agreements  and  contracts 
have  been  made,  use  them  as  the  basis  for  a  TRA.  Otherwise,  identify  and  assess  any  critical  technol¬ 
ogy  bid  in  any  of  the  Technology  Development  proposals. 
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3.2.3  Data  Collection 


Relevant  data  and  information  are  needed  to  assess  the  TRL  for  each  CTE.  The 
process  of  collecting  and  organizing  the  material  for  each  CTE  should  begin  as  early  as 
possible.  Figure  3-2  shows  this  process  as  being  concurrent  with  CTE  identification.  Data 
collection  should  be  complete  when  the  CTEs  have  been  finalized.  The  assessment  pro¬ 
cess  will  be  disrupted  and  delayed  if  relevant  data  are  not  easily  accessible  at  the  time 
these  data  are  needed. 

Key  Player  Roles  and  Responsibilities  in  Data  Collection 

•  PM.  Compile  component  or  subsystem  test  descriptions,  environments,  and 
results  in  the  context  of  the  system’s  functional  needs.  Any  other  analyses 
and  information  necessary  to  assess  and  rationalize  the  maturity  of  the  CTEs 
should  also  be  included. 

3.2.4  CTEs  Coordinated 

At  this  point,  any  disagreements  on  identifying  CTEs  should  be  resolved  within 
the  Component.  A  DUSD(S&T)  agreement  on  the  CTEs  should  also  be  obtained  so  that 
any  concerns  can  be  raised  early  and  addressed  in  a  timely  manner. 

Key  Player  Roles  and  Responsibilities  in  Coordinating  CTEs 

•  PM.  After  reviewing  the 
recommendations  of  the 
independent  panel,  sub¬ 
mit  the  final  CTE  iden¬ 
tification  data  to  the 
Component  S&T  Exec¬ 
utive  and  request  a 
TRA.  An  information 
copy  should  be  sent  to  the  DUSD(S&T)  for  ACAT  ID  and  ACAT  IAM  pro¬ 
grams.  As  part  of  this  submission,  explain  the  function  of  each  CTE  at  the 
component,  subsystem,  and  system  levels  and  describe  the  rationale  and  cri¬ 
teria  for  declaring  this  technology  critical.  Also,  briefly  explain  the  process 
and  criteria  used  to  eliminate  the  CTE  candidates  that  were  not  judged  to  be 
critical.  Provide  any  additional  information  requested  by  the  Component 
S&T  Executive  or  the  DUSD(S&T). 

•  Component  S&T  Executive.  Review  the  CTEs  and  coordinate  with  the  PM 
and  the  DUSD(S&T)  on  any  additions  or  deletions  and  on  any  additional 
information  needed  for  the  TRA. 


Best  Practice 

CTE  identification  data  should  include 
a  brief  description  of  the  rationale 
for  declaring  a  CTE  to  be  critical  and 
of  the  process  and  criteria  for  elimi¬ 
nating  a  CTE  candidate. 
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3.3  ASSESSING  CTE  READINESS29 


Figure  3-3  is  extracted  from  the  bottom  of  Figure  3-1.  It  shows  a  possible  sched¬ 
ule  of  activities  to  assess  CTE  readiness  as  a  continuation  of  the  schedule  shown  in  Fig¬ 
ure  3-2.  The  following  subsections  describe  the  activities  for  each  line  in  Figure  3-3. 


West.  West.  West  West.  West  West.  West.  West  West.  West  'West.  West  West.  Weet. 
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TRA  Performed 
TRA  Coordination 

DUSD(S&T)  TRA  Review  &  Evaluation 

Independent  TRA  (if  necessary) 

Evaluation  Memo 

Milestone  Review 


£ 


i 


Figure  3-3.  Representative  Schedule  for  Assessing  CTE  Readiness 


3.3.1  TRA  Performed 

Depending  on  the  number  of  CTEs  to  be  assessed  and  the  complexity  of  the  sys¬ 
tem,  completing  the  process  may  require  several  months.  Given  all  the  data,  assessing  the 
maturity  of  a  technology  does  not  take  very  long.  However,  the  amount  of  time  needed  to 
complete  the  process  is  also  a  function  of  iterative  data-collection  efforts,  obtaining 
answers  to  questions,  scheduling  meetings,  and  so  forth. 


Key  Player  Roles  and  Responsibilities  in  Performing  the  TRA 

•  Component  S&T  Executive.  Conduct  the  TRA  in  accordance  with  Compo¬ 
nent  guidelines  and  procedures.  Appoint  and  train  an  independent  team30  to 
make  the  assessments.  Training  should  include  an  overview  of  the  system,  an 
overview  of  the  TRA  process,  criteria  for  identifying  CTEs,  and  examples 
and  instructions  for  the  application  of  the  TRLs.  Keep  DUSD(S&T) 
informed,  as  appropriate.31 


29  See  Appendix  C  for  more  details. 

30  This  may  or  may  not  be  the  same  team  used  to  provide  a  recommendation  on  CTEs. 

31  The  Component  S&T  Executive  is  not  required  to  agree  to  any  monitoring  or  participation  beyond 
oversight;  however,  greater  DUSD(S&T)  involvement  facilitates  quicker  concurrence.  The 
DUSD(S&T)  AO  should  review  the  CTEs  and  the  identification  process,  negotiate  any  perceived  defi¬ 
ciencies,  and  provide  oversight  on  the  overall  process  while  the  Component  TRA  is  conducted.  The 
AO  should  coordinate  with  the  Component  S&T  Executive  to  determine  to  what  extent  he/she  and/or 
technology  specialists  designated  by  the  DUSD(S&T)  could  or  should  monitor  or  participate  in  the 
Component  TRA. 
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•  Independent  Team.  Assess 

the  TRL  for  all  CTEs.  Pre¬ 
pare  the  TRA  for  submis¬ 
sion  once  the  assessment  is 
made.  The  TRA  assessment 
should  explain  the  function  of  each  CTE  at  the  component,  subsystem,  and  system 
level.  It  should  also  describe  the  rationale  and  criteria  for  declaring  these 
technologies  to  be  critical  and  for  declaring  any  candidate  technology  not  to  be 
critical.32  The  TRA  should  also  include  the  basis  for  the  assessment.  Evidence 
could  include  records  of  tests  or  applications  of  the  technology,  technical  papers, 
reports,  presentations,  and  so  forth.  Explain  how  the  material  was  used  or 
interpreted  to  make  the  assessment.  The  TRA  at  Milestone  C  should  highlight  the 
assessment  of  CTEs  that  did  not  attain  a  TRL  6  at  Milestone  B  and  additional 
CTEs  that  were  identified  during  SDD— especially  any  manufacturing  CTEs.  For 
MAIS  acquisition  and  software-intensive  systems  at  Milestone  C,  describe  the 
results  of  DT&E  for  all  CTEs. 

The  Component  should  use  TRLs  to  communicate  TRA  findings.  Tables  3-1 
through  3-4  show  TRL  definitions,  descriptions,  and  supporting  information  for  hard¬ 
ware,  software,  and  manufacturing  technologies.33 

3.3.2  TRA  Coordination 

The  TRA  should  be  submitted  to  the  DUSD(S&T)  according  to  the  agreed-upon 
schedule— normally,  at  least  6  weeks  before  a  scheduled  Milestone  B  or  Milestone  C. 
Allow  at  least  2  weeks  for  the  coordination  process  within  the  Component  before  TRA 
submission.  See  Figure  3-3. 

The  coordination  should  take  into  account  more  than  agreement  on  the  value  of 
the  TRL.  The  effect  of  immature  technology  on  programmatics  is  an  even  more  important 
consideration.  DoD  policy  on  technology  maturation  is  clear.  One  of  the  entry 


Best  Practice 

The  TRA  should  include  the  CTE  iden¬ 
tification  rationale  and  the  basis  for 
the  assessment. 


32  This  represents  the  minimum  requirement  for  a  TRA  if  no  technology  was  identified  as  being  critical 
using  the  criteria  described  in  Subsection  3.2,  as  supplemented  by  Appendix  D. 

33  See  Appendix  H  for  a  discussion  of  biomedical  TRLs  and  Appendix  I  for  a  discussion  of  Manufac¬ 
turing  Readiness  Levels  (MRLs). 
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Table  3-1.  Hardware  TRL  Definitions,  Descriptions,  and  Supporting  Information 
(Source:  Defense  Acquisition  Guidebook) 


TRL 

Definition 

Description 

Supporting  Information 

1 

Basic  principles 
observed  and 
reported. 

Lowest  level  of  technology  readi¬ 
ness.  Scientific  research  begins 
to  be  translated  into  applied 
research  and  development  (R&D). 
Examples  might  include  paper 
studies  of  a  technology’s  basic 
properties. 

Published  research  that  identifies  the  prin¬ 
ciples  that  underlie  this  technology.  Refer¬ 
ences  to  who,  where,  when. 

2 

Technology  con¬ 
cept  and/or  appli¬ 
cation  formulated. 

Invention  begins.  Once  basic 
principles  are  observed,  practical 
applications  can  be  invented. 
Applications  are  speculative,  and 
there  may  be  no  proof  or  detailed 
analysis  to  support  the  assump¬ 
tions.  Examples  are  limited  to 
analytic  studies. 

Publications  or  other  references  that  out¬ 
line  the  application  being  considered  and 
that  provide  analysis  to  support  the 
concept. 

3 

Analytical  and 
experimental 
critical  function 
and/or  character¬ 
istic  proof  of 
concept. 

Active  R&D  is  initiated.  This 
includes  analytical  studies  and 
laboratory  studies  to  physically 
validate  the  analytical  predictions 
of  separate  elements  of  the  tech¬ 
nology.  Examples  include 
components  that  are  not  yet  inte¬ 
grated  or  representative. 

Results  of  laboratory  tests  performed  to 
measure  parameters  of  interest  and  com¬ 
parison  to  analytical  predictions  for  critical 
subsystems.  References  to  who,  where, 
and  when  these  tests  and  comparisons 
were  performed. 

4 

Component 
and/or  bread¬ 
board  validation 
in  a  laboratory 
environment. 

Basic  technological  components 
are  integrated  to  establish  that 
they  will  work  together.  This  is 
relatively  “low  fidelity”  compared 
with  the  eventual  system.  Exam¬ 
ples  include  integration  of  “ad 
hoc”  hardware  in  the  laboratory. 

System  concepts  that  have  been  consid¬ 
ered  and  results  from  testing  laboratory- 
scale  breadboard(s).  References  to  who 
did  this  work  and  when.  Provide  an  esti¬ 
mate  of  how  breadboard  hardware  and 
test  results  differ  from  the  expected  system 
goals. 

5 

Component  and / 
or  breadboard 
validation  in  a 
relevant 
environment. 

Fidelity  of  breadboard  technology 
increases  significantly.  The  basic 
technological  components  are 
integrated  with  reasonably  realis¬ 
tic  supporting  elements  so  they 
can  be  tested  in  a  simulated  envi¬ 
ronment.  Examples  include  “high- 
fidelity”  laboratory  integration  of 
components. 

Results  from  testing  a  laboratory  bread¬ 
board  system  are  integrated  with  other 
supporting  elements  in  a  simulated  opera¬ 
tional  environment.  How  does  the  “relevant 
environment”  differ  from  the  expected 
operational  environment?  How  do  the  test 
results  compare  with  expectations?  What 
problems,  if  any,  were  encountered?  Was 
the  breadboard  system  refined  to  more 
nearly  match  the  expected  system  goals? 
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Table  3-1.  Hardware  TRL  Definitions,  Descriptions,  and  Supporting  Information 
(Source:  Defense  Acquisition  Guidebook)  (Continued) 


TRL 

Definition 

Description 

Supporting  Information 

6 

System/subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment. 

Representative  model  or  proto¬ 
type  system,  which  is  well  be¬ 
yond  that  of  TRL  5,  is  tested  in  a 
relevant  environment.  Repre¬ 
sents  a  major  step  up  in  a  tech¬ 
nology’s  demonstrated 
readiness.  Examples  include 
testing  a  prototype  in  a  high- 
fidelity  laboratory  environment 
or  in  a  simulated  operational 
environment. 

Results  from  laboratory  testing 
of  a  prototype  system  that  is 
near  the  desired  configuration 
in  terms  of  performance, 
weight,  and  volume.  How  did 
the  test  environment  differ  from 
the  operational  environment? 
Who  performed  the  tests?  How 
did  the  test  compare  with 
expectations?  What  problems, 
if  any,  were  encountered? 

What  are/were  the  plans, 
options,  or  actions  to  resolve 
problems  before  moving  to  the 
next  level? 

7 

System  prototype  demon¬ 
stration  in  an  operational 
environment. 

Prototype  near  or  at  planned 
operational  system.  Represents 
a  major  step  up  from  TRL  6  by 
requiring  demonstration  of  an 
actual  system  prototype  in  an 
operational  environment  (e.g.,  in 
an  aircraft,  in  a  vehicle,  or  in 
space).  Examples  include 
testing  the  prototype  in  a  test 
bed  aircraft. 

Results  from  testing  a  proto¬ 
type  system  in  an  operational 
environment.  Who  performed 
the  tests?  How  did  the  test 
compare  with  expectations? 

What  problems,  if  any,  were 
encountered?  What  are/were 
the  plans,  options,  or  actions  to 
resolve  problems  before 
moving  to  the  next  level? 

8 

Actual  system  completed 
and  qualified  through  test 
and  demonstration. 

Technology  has  been  proven  to 
work  in  its  final  form  and  under 
expected  conditions.  In  almost 
all  cases,  this  TRL  represents 
the  end  of  true  system  develop¬ 
ment.  Examples  include  devel¬ 
opmental  test  and  evaluation  of 
the  system  in  its  intended  wea¬ 
pon  system  to  determine  if  it 
meets  design  specifications. 

Results  of  testing  the  system  in 
its  final  configuration  under  the 
expected  range  of  environ¬ 
mental  conditions  in  which  it 
will  be  expected  to  operate. 
Assessment  of  whether  it  will 
meet  its  operational  require¬ 
ments.  What  problems,  if  any, 
were  encountered?  What  are/ 
were  the  plans,  options,  or 
actions  to  resolve  problems 
before  finalizing  the  design? 

9 

Actual  system  proven 
through  successful  mission 
operations. 

Actual  application  of  the  tech¬ 
nology  in  its  final  form  and  under 
mission  conditions,  such  as 
those  encountered  in  opera¬ 
tional  test  and  evaluation 
(OT&E).  Examples  include  using 
the  system  under  operational 
mission  conditions. 

OT&E  reports. 
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Table  3-2.  Additional  Definitions  of  TRL  Descriptive  Terms 
(Source:  Defense  Acquisition  Guidebook) 


Term 

Definition 

Breadboard 

Integrated  components  that  provide  a  representation  of  a  system/ 
subsystem  and  that  can  be  used  to  determine  concept  feasibility 
and  to  develop  technical  data.  Typically  configured  for  laboratory 
use  to  demonstrate  the  technical  principles  of  immediate  interest. 

May  resemble  final  system/subsystem  in  function  only. 

High  Fidelity 

Addresses  form,  fit,  and  function.  A  high-fidelity  laboratory  environ¬ 
ment  would  involve  testing  with  equipment  that  can  simulate  and 
validate  all  system  specifications  within  a  laboratory  setting. 

Low  Fidelity 

A  representative  of  the  component  or  system  that  has  limited  ability 
to  provide  anything  but  first-order  information  about  the  end  product. 
Low-fidelity  assessments  are  used  to  provide  trend  analysis. 

Model 

A  functional  form  of  a  system,  generally  reduced  in  scale,  near  or  at 
operational  specification.  Models  will  be  sufficiently  hardened  to 
allow  demonstration  of  the  technical  and  operational  capabilities 
required  of  the  final  system. 

Operational  Environment 

Environment  that  addresses  all  the  operational  requirements  and 
specifications  required  of  the  final  system  to  include  platform/ 
packaging. 

Prototype 

A  physical  or  virtual  model  used  to  evaluate  the  technical  or  manu¬ 
facturing  feasibility  or  military  utility  of  a  particular  technology  or 
process,  concept,  end  item,  or  system. 

Relevant  Environment 

Testing  environment  that  simulates  the  key  aspects  of  the  opera¬ 
tional  environment. 

Simulated  Operational  Environment 

Either  (1)  a  real  environment  that  can  simulate  all  the  operational 
requirements  and  specifications  required  of  the  final  system  or  (2)  a 
simulated  environment  that  allows  for  testing  of  a  virtual  prototype. 
Used  in  either  case  to  determine  whether  a  developmental  system 
meets  the  operational  requirements  and  specifications  of  the  final 
system. 

Table  3-3.  Software  TRL  Definitions,  Descriptions,  and  Supporting  Information 
(Source:  IT  TRL  Working  Group  Minutes,  November  9,  2004) 


TRL 

Definition 

Description 

Supporting  Information 

1 

Basic  principles  observed 
and  reported. 

Lowest  level  of  software  technol¬ 
ogy  readiness.  A  new  software 
domain  is  being  investigated  by 
the  basic  research  community. 

This  level  extends  to  the  devel¬ 
opment  of  basic  use,  basic  prop¬ 
erties  of  software  architecture, 
mathematical  formulations,  and 
general  algorithms. 

Basic  research  activities, 
research  articles,  peer-reviewed 
white  papers,  point  papers,  early 
lab  model  of  basic  concept  may 
be  useful  for  substantiating  the 
TRL  level. 
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Table  3-3.  Software  TRL  Definitions,  Descriptions,  and  Supporting  Information 
(Source:  IT  TRL  Working  Group  Minutes,  November  9,  2004)  (Continued) 


TRL 

Definition 

Description 

Supporting  Information 

2 

Technology  concept 
and/or  application 
formulated. 

Once  basic  principles  are 
observed,  practical  applications 
can  be  invented.  Applications  are 
speculative,  and  there  may  be  no 
proof  or  detailed  analysis  to  sup¬ 
port  the  assumptions.  Examples 
are  limited  to  analytic  studies 
using  synthetic  data. 

Applied  research  activities,  ana¬ 
lytic  studies,  small  code  units, 
and  papers  comparing 
competing  technologies. 

3 

Analytical  and  experi¬ 
mental  critical  function 
and/or  characteristic  proof 
of  concept. 

Active  R&D  is  initiated.  The  level 
at  which  scientific  feasibility  is 
demonstrated  through  analytical 
and  laboratory  studies.  This  level 
extends  to  the  development  of 
limited  functionality  environments 
to  validate  critical  properties  and 
analytical  predictions  using  non- 
integrated  software  components 
and  partially  representative  data. 

Algorithms  run  on  a  surrogate 
processor  in  a  laboratory  envi¬ 
ronment,  instrumented  compo¬ 
nents  operating  in  laboratory 
environment,  laboratory  results 
showing  validation  of  critical 
properties. 

4 

Module  and/or  subsystem 
validation  in  a  laboratory 
environment  (i.e.,  software 
prototype  development 
environment). 

Basic  software  components  are 
integrated  to  establish  that  they 
will  work  together.  They  are  rela¬ 
tively  primitive  with  regard  to 
efficiency  and  robustness  com¬ 
pared  with  the  eventual  system. 
Architecture  development  initi¬ 
ated  to  include  interoperability, 
reliability,  maintainability,  exten¬ 
sibility,  scalability,  and  security 
issues.  Emulation  with  current/ 
legacy  elements  as  appropriate. 
Prototypes  developed  to  dem¬ 
onstrate  different  aspects  of 
eventual  system. 

Advanced  technology  develop¬ 
ment,  stand-alone  prototype 
solving  a  synthetic  full-scale 
problem,  or  standalone  proto¬ 
type  processing  fully  represen¬ 
tative  data  sets. 

5 

Module  and/or  subsystem 
validation  in  a  relevant 
environment. 

Level  at  which  software  tech¬ 
nology  is  ready  to  start  integra¬ 
tion  with  existing  systems.  The 
prototype  implementations  con¬ 
form  to  target  environment/ 
interfaces.  Experiments  with 
realistic  problems.  Simulated 
interfaces  to  existing  systems. 
System  software  architecture 
established.  Algorithms  run  on  a 
processor(s)  with  characteristics 
expected  in  the  operational 
environment. 

System  architecture  diagram 
around  technology  element  with 
critical  performance  require¬ 
ments  defined.  Processor  selec¬ 
tion  analysis,  Simulation/ 
Stimulation  (Sim/Stim)  Labora¬ 
tory  buildup  plan.  Software 
placed  under  configuration  man¬ 
agement.  COTS/GOTS  in  the 
system  software  architecture  are 
identified. 
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Table  3-3.  Software  TRL  Definitions,  Descriptions,  and  Supporting  Information 
(Source:  IT  TRL  Working  Group  minutes,  November  9,  2004)  (Continued) 


TRL 

Definition 

Description 

Supporting  Information 

6 

Module  and/or  subsystem 
validation  in  a  relevant 
end-to-end  environment. 

Level  at  which  the  engineering 
feasibility  of  a  software  technol¬ 
ogy  is  demonstrated.  This  level 
extends  to  laboratory  prototype 
implementations  on  full-scale 
realistic  problems  in  which  the 
software  technology  is  partially 
integrated  with  existing  hard¬ 
ware/software  systems. 

Results  from  laboratory  testing 
of  a  prototype  package  that  is 
near  the  desired  configuration  in 
terms  of  performance,  including 
physical,  logical,  data,  and  secu¬ 
rity  interfaces.  Comparisons 
between  tested  environment  and 
operational  environment  analyti¬ 
cally  understood.  Analysis  and 
test  measurements  quantifying 
contribution  to  system-wide 
requirements  such  as  through¬ 
put,  scalability,  and  reliability. 
Analysis  of  human-computer 
(user  environment)  begun. 

7 

System  prototype  demon¬ 
stration  in  an  operational 
high-fidelity  environment. 

Level  at  which  the  program  fea¬ 
sibility  of  a  software  technology  is 
demonstrated.  This  level  extends 
to  operational  environment  proto¬ 
type  implementations  where  criti¬ 
cal  technical  risk  functionality  is 
available  for  demonstration  and  a 
test  in  which  the  software  tech¬ 
nology  is  well  integrated  with 
operational  hardware/software 
systems. 

Critical  technological  properties 
are  measured  against  require¬ 
ments  in  a  simulated  operational 
environment. 

8 

Actual  system  completed 
and  mission  qualified 
through  test  and  demon¬ 
stration  in  an  operational 
environment. 

Level  at  which  a  software  tech¬ 
nology  is  fully  integrated  with 
operational  hardware  and  soft¬ 
ware  systems.  Software  develop¬ 
ment  documentation  is  complete. 
All  functionality  tested  in  simul¬ 
ated  and  operational  scenarios. 

Published  documentation  and 
product  technology  refresh  build 
schedule.  Software  resource 
reserve  measured  and  tracked. 

9 

Actual  system  proven 
through  successful  mis¬ 
sion-proven  operational 
capabilities. 

Level  at  which  a  software  tech¬ 
nology  is  readily  repeatable  and 
reusable.  The  software  based  on 
the  technology  is  fully  integrated 
with  operational  hardware/soft¬ 
ware  systems.  All  software 
documentation  verified.  Suc¬ 
cessful  operational  experience. 
Sustaining  software  engineering 
support  in  place.  Actual  system. 

Production  configuration  man¬ 
agement  reports.  Technology 
integrated  into  a  reuse  “wizard”; 
out-year  funding  established  for 
support  activity. 
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Table  3-4.  Manufacturing  Technology  TRL  Definitions, 
Descriptions,  and  Supporting  Information 
[Source:  Joint  Defense  Manufacturing  Technology  Panel  (JDMPT) 
Manufacturing  Readiness  Level  Subgroup ] 


TRL 

Definition 

Description 

Supporting  Information 

1 

Basic  principles 
observed  and 
reported. 

NA 

2 

Technology  con¬ 
cept  and/or  appli¬ 
cation  formulated. 

NA 

3 

Analytical  and 
experimental  criti¬ 
cal  function  and / 
or  characteristic 
proof  of  concept. 

NA 

4 

Component  and / 
or  breadboard 
validation  in  a 
laboratory 
environment. 

The  new  technology  has 
been  demonstrated  in  a 
laboratory  environment  on 
simple  design  parts  using 
similar  types  of  materials 
that  would  be  used  in  the 
intended  application. 

This  is  the  lowest  level  of  production  readiness. 
Component  technologies  must  have  matured  to 
at  least  TRL  4.  At  this  point,  few  requirements 
have  been  validated,  and  there  will  be  a  large 
number  of  engineering/design  changes.  Com¬ 
ponent  physical  and  functional  interfaces  have 
not  been  defined.  Materials,  machines,  and 
tooling  have  been  demonstrated  in  a  laboratory 
environment.  Inspection  and  test  equipment 
have  been  demonstrated  in  a  laboratory  envi¬ 
ronment.  Manufacturing  cost  drivers  are  identi¬ 
fied.  Producibility  assessments  have  been 
initiated. 

5 

Component  and / 
or  breadboard 
validation  in  a 
relevant 
environment. 

The  new  technology  has 
been  demonstrated  in  a 
laboratory  environment  on 
design  parts  of  the  same 
level  of  complexity  and 
using  the  same  types  of 
materials  that  would  be 
used  in  the  intended 
application. 

Component  technologies  must  have  matured  to 
at  least  TRL  5.  At  this  point,  all  requirements 
have  not  been  validated,  and  there  will  be  sig¬ 
nificant  engineering/design  changes.  Compo¬ 
nent  physical  and  functional  interfaces  have  not 
been  defined.  Materials,  machines,  and  tooling 
have  been  demonstrated  in  a  relevant  manufac¬ 
turing  environment,  but  most  manufacturing 
processes  and  procedures  are  in  development 
(or  ManTech  initiatives  are  ongoing).  Inspection 
and  test  equipment  have  been  demonstrated  in 
a  laboratory  environment.  Production  cost 
drivers/goals  are  analyzed.  System-level  DTC 
goals  are  set.  Producibility  assessments  are 
ongoing. 
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Table  3-4.  Manufacturing  Technology  TRL  Definitions, 
Descriptions,  and  Supporting  Information 
(Source:  JDMTP  Manufacturing  Readiness  Level  Subgroup)  (Continued) 


TRL 

Definition 

Description 

Supporting  Information 

6 

System/subsyste 
m  model  or  pro¬ 
totype  demonstra¬ 
tion  in  a  relevant 
environment. 

The  new  technology  has 
been  demonstrated  in  a  pre- 
production  environment  on 
design  parts  of  the  same 
level  of  complexity  and 
using  the  same  types  of 
materials  that  would  be 
used  in  the  intended  appli¬ 
cation.  Appropriate  quality 
levels  have  been  achieved. 

During  the  prototype  demonstration,  phase 
requirements  are  validated  and  defined.  How¬ 
ever,  there  will  still  be  many  engineering/design 
changes,  and  the  physical  and  functional  inter¬ 
faces  are  not  yet  fully  defined.  Component  tech¬ 
nologies  must  have  matured  to  at  least  TRL  6. 
Raw  materials  are  initially  demonstrated  in 
relevant  manufacturing  environment.  Similar 
processes  and  procedures  have  been  demon¬ 
strated  in  a  relevant  manufacturing  environ¬ 
ment.  At  this  point,  there  are  likely  major 
investments  required  for  machines  and  tooling. 
Inspection  and  test  equipment  should  be  under 
development.  Producibility  assessments  are 
ongoing,  and  trade  studies  have  been  con¬ 
ducted.  A  production  Cost  Reduction  Plan  is 
developed.  Production  goals  are  set. 

7 

System  prototype 
demonstration  in 
an  operational 
environment. 

The  new  technology  has 
been  demonstrated  in  a 
relevant  production  envi¬ 
ronment  on  design  parts  of 
the  same  level  of  complexity 
and  using  the  same  types  of 
materials  that  would  be 
used  in  the  intended  appli¬ 
cation.  Appropriate  quality 
and  throughput  levels  have 
been  achieved. 

Component  technologies  must  have  matured  to 
at  least  TRL  7.  At  this  point,  engineering/design 
changes  should  decrease.  Physical  and  func¬ 
tional  interfaces  should  be  clearly  defined.  All 
raw  materials  are  in  production  and  available  to 
meet  planned  LRIP  schedule.  Pilot  line  manu¬ 
facturing  processes  and  procedures  set  up  and 
under  test.  Processes  and  procedures  are  not 
yet  proven  or  under  control.  During  this  phase, 
initial  producibility  improvements  should  be 
underway.  DTC  estimates  are  less  than 

125  percent  of  goals.  Detailed  production  esti¬ 
mates  are  established. 

8 

Actual  system 
completed  and 
qualified  through 
test  and 
demonstration. 

The  new  technology  has 
been  demonstrated  in  a  pilot 
production  environment  on 
production-representative 
parts  of  the  same  level  of 
complexity  and  using  the 
same  types  of  materials  that 
would  be  used  in  the 
intended  application.  Appro¬ 
priate  quality  and  throughput 
levels  have  been  achieved. 
Process  has  been  proven 
and  is  under  control  for 

LRIP. 

Component  technologies  must  have  matured  to 
at  least  TRL  8.  At  this  point,  engineering/design 
changes  should  decrease  significantly.  Physical 
and  functional  interfaces  should  be  clearly 
defined.  All  raw  materials  are  in  production  and 
available  to  meet  the  planned  LRIP  schedule. 
Manufacturing  processes  and  procedures  have 
been  proven  on  the  pilot  line,  are  under  control, 
and  are  ready  for  LRIP.  During  this  phase,  initial 
producibility  risk  assessments  should  be  com¬ 
pleted.  Production  cost  estimates  meet  DTC 
goals. 
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Table  3-4.  Manufacturing  Technology  TRL  Definitions, 
Descriptions,  and  Supporting  Information 
(Source:  JDMTP  Manufacturing  Readiness  Level  Subgroup)  (Continued) 


TRL 

Definition 

Description 

Supporting  Information 

9 

Actual  system 
proven  through 
successful  mis¬ 
sion  operations. 

The  new  technology  has 
been  demonstrated  in  an 

LRIP  environment  on 
intended  parts  and  using  the 
intended  types  of  materials. 
Process  has  been  proven 
and  under  control  for 
production. 

During  LRIP,  all  systems  engineering/design 
requirements  should  be  met  and  there  should 
only  be  minimal  system  engineering/design 
changes.  Technologies  must  have  matured  to 
at  least  TRL  9.  Materials  are  in  production  and 
available  to  meet  planned  production  sched¬ 
ules.  Manufacturing  processes  and  procedures 
are  established  and  controlled  in  production  to 
three-sigma  or  some  other  appropriate  quality 
level.  Machines,  tooling  and  inspection  and  test 
equipment  deliver  three-sigma  or  some  other 
appropriate  quality  level  in  production.  Produc¬ 
tion  risk  monitoring  is  ongoing.  LRIP  actual 
costs  meet  estimates. 

criteria  for  SDD  is  that  all  CTEs  should  have  been  demonstrated  in  a  relevant  environ¬ 
ment.  Therefore,  if  the  TRA  indicates  otherwise,  the  Component  has  three  choices:34 

1.  Request  a  delay  of  the  Milestone  review  until  all  CTEs  are  at  the  requisite 
maturity  level 

2.  Use  alternative,  mature  technologies  in  the  program 

3.  As  a  last  resort,  carry  immature  technologies  into  the  Milestone  review  and 
submit  a  Technology  Maturation  Plan  (TMP)  along  with  the  TRA. 

Key  Player  Roles  and  Responsibilities  in  TRA  Coordination 

•  Component  S&T  Executive.  For  ACAT  ID  and  ACAT  IAM  programs, 
approve  the  TRA,  accept  responsibility  for  its  accuracy  based  on  established 
standards  and  expectations  for  quality,  submit  (e.g.,  sign  a  forwarding 
memorandum)  the  TRA  to  the  CAE  or  Agency  Head  and,  at  the  same  time, 
send  an  information  copy  to  the  DUSD(S&T). 

•  Component  Acquisition  Executive  or  Agency  Head.  For  ACAT  ID  and 
ACAT  IAM  programs,  submit  a  report  to  the  DUSD(S&T),  with  an  assessed 
TRL  for  each  CTE.  This  report  can  consist  of  a  cover  letter  or  memorandum 
endorsing  the  Component  TRA  and  officially  transmitting  that  TRA.  The 
report  endorses  an  agreement  with  the  assessed  readiness  levels  and,  if  any  of 
the  CTEs  are  immature,  a  commitment  to  a  TMP. 


34  See  Section  5  for  a  more  complete  discussion  of  this  topic. 
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3.3.3  DUSD(S&T)  TRA  Review  and  Evaluation 

The  DUSD(S&T)  evaluates  the  Component  TRA  in  cooperation  with  the  Compo¬ 
nent  S&T  Executive  and  the  PM.  An  AO,  designated  by  the  DUSD(S&T),  will  normally 
lead  the  evaluation  effort.35  After  an  initial  evaluation,  the  AO  can  either  concur  or 
request  that  revisions  be  made.  In  the  latter  case,  the  TRA  will  be  updated  and  returned  to 
the  AO  for  further  review. 

If  the  DUSD(S&T)  does  not  concur  with  the  findings  of  the  Component  TRA,  an 
independent  technical  assessment  can  be  conducted.  This  independent  assessment  should 
be  a  positive  contribution  to  the  acquisition  program.  For  example,  it  could  result  in  a 
revised,  more  realistic  schedule,  in  the  use  of  an  alternative  technology,  or  in  a  revised, 
evolutionary  acquisition  strategy.  The  independent  assessment  should  be  conducted  as 
quickly  as  possible— whether  this  requires  one  day  or  several  months.  Typically,  the 
Component  funds  the  independent  assessment. 

The  AO  prepares  a  memorandum  for  DUSD(S&T)  signature.  This  memorandum 
contains  the  evaluation  results  of  the  Component  TRA  and  of  the  independent  technical 
assessment  (if  an  independent  technical  assessment  was  conducted).  It  indicates  either 
concurrence  or  concurrence  with  reservations  concerning  the  findings  of  the  Component 
TRA,  or  it  contains  the  findings  of  the  independent  technical  assessment.  If  the  AO 
deems  any  CTE  to  be  insufficiently  mature  for  the  coming  milestone,  he/she  informs  the 
Component  S&T  Executive  and  the  PM  so  that  all  involved  have  an  opportunity  to  reach 
agreement  on  appropriate  action.  The  memorandum  is  sent  to  the  Overarching  Integrated 
Product  Team  (OIPT)  and  the  Defense  Acquisition  Board  (DAB)  or  to  the  Information 
Technology  Overarching  Integrated  Product  Team  (IT  OIPT)  and  the  Information  Tech¬ 
nology  Acquisition  Board  (ITAB).  The  evaluation  memorandum  should  be  signed  at  least 
15  days  before  a  Milestone  B  or  Milestone  C  review  meeting.36  This  memo  is  forwarded 
to  the  MDA  and,  if  there  is  nonconcurrence,  to  the  OIPT/ITOIPT  and  the  DAB/IT AB. 
Concurrence  with  reservation  is  handled  on  a  case-by-case  basis. 


35  The  AO  calls  for  assistance,  as  necessary,  to  obtain  a  competent  assessment  of  the  CTEs  and  to  deter¬ 
mine  whether  all  the  CTEs  have  been  identified. 

36  If  this  15-day  window  is  not  possible,  the  date  of  the  review  meeting  should  be  reconsidered  so  the 
OIPT  and  DAB  members  or  the  IT  OIPT  and  ITAB  members  have  ample  time  to  review  all  the 
relevant  information.  As  appropriate,  the  memorandum  should  address  recommendations  to  the  MDA 
for  issues  to  be  raised  at  the  Milestone  review  and  for  items  to  be  included  in  the  ADM. 
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SECTION  4. 
SUBMITTING  A  TRA 


4.1  SKELETAL  TEMPLATE  FOR  A  TRA  SUBMISSION 

The  following  outline  is  a  skeletal  template  for  anticipated  TRA  submissions: 

1.0  Purpose  of  This  Document 
2.0  Program  Overview 

2.1  Program  Objective 

2.2  Program  Description 

2.3  System  Description 

3.0  Technology  Readiness  Assessment 

3.1  Process  Description 

3.2  CTEs 

3.3  Assessment  of  Maturity 

3.3.1  First  CTE  or  Category  of  Technology 

3.3.2  Next  CTE  or  Category  of  Technology 

3.4  Summary  of  TRLs  by  Technology 

4.0  Conclusion 
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4.2  ANNOTATED  TEMPLATE  FOR  A  TRA  SUBMISSION 


37 


The  following  outline  is  an  annotated  version  of  the  TRA  template.  37 

1.0  Purpose  of  This  Document 

Should  be  short  and  should  give  the  program  name,  the  system  name  if  dif¬ 
ferent  from  the  program  name,  and  the  milestone  or  other  decision  point  for  which 
the  TRA  was  performed.  For  example,  “This  document  presents  an  independent 
Technology  Readiness  Assessment  (TRA)  for  the  UH-60M  helicopter  program  in 
support  of  the  Milestone  B  decision.  The  TRA  was  performed  at  the  direction  of 
the  Army  S&T  Executive.” 

2.0  Program  Overview 

2.1  Program  Objective 

States  what  the  program  is  trying  to  achieve  (e.g.,  new  capability,  improved 
capability,  lower  procurement  cost,  reduced  maintenance  or  manning,  and  so 
forth).  Refers  to  the  Capability  Development  Document  (CDD)  (for  Milestone  B) 
or  the  Capability  Production  Document  (CPD)  (for  Milestone  C)  that  documents 
the  program  objectives. 

2.2  Program  Description 

Briefly  describes  the  program,  not  the  system.  Does  the  program  provide  a 
new  system  or  a  modification  to  an  existing  operational  system?  Is  it  an  evolu¬ 
tionary  acquisition  program?  If  so,  what  capabilities  will  be  realized  in  Incre¬ 
ment  1?  When  is  the  initial  operational  capability  (IOC)?  Does  it  have  multiple 
competing  prime  contractors?  Into  what  architecture  does  it  fit?  Is  it  a  system-of- 
systems?  Does  its  success  depend  on  the  success  of  other  acquisition  programs? 

2.3  System  Description 

Describes  the  overall  system,  the  major  subsystems,  and  components,  as 
necessary,  to  give  an  understanding  of  what  is  being  developed  and  to  show  what 
is  new,  unique,  or  special  about  it.  This  should  include  the  systems,  components, 
and  technologies  that  will  later  be  declared  Critical  Technology  Elements  (CTEs) 
Describes  how  the  system  works  (if  this  is  not  obvious). 


People  who  directly  contribute  to  the  generation  of  TRAs  can  obtain  examples  of  TRA  reports  from 
Mr.  Jack  Taylor.  Mr.  Taylor’s  contact  information  is  as  follows:  DDR&E,  1777  North  Kent  Street, 
Rosslyn,  VA  22209.  Phone  number:  703-588-7405;  e-mail  address:  jack.taylor@osd.mil. 
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3.0  Technology  Readiness  Assessment  (TRA) 

3.1  Process  Description 

Tells  who  led  the  TRA  and  what  organizations  or  individuals  performed  the 
TRA.  Identifies  the  special  expertise  of  participating  organizations  or  individuals. 
This  should  establish  the  competence  and  the  independence  of  the  TRA.  In  this 
context,  “independence”  means  that  the  assessors  are  not  unduly  influenced  by  the 
opinions  of  the  developers  (government  or  industry).  Usually,  the  PM  or  the  Sys¬ 
tem  Program  Office  (SPO)  will  provide  most  of  the  data  and  other  information 
that  form  the  basis  of  a  TRA.  Nevertheless,  the  assessment  should  be  independent 
of  the  PM  or  SPO. 

Tells  how  CTEs  were  identified  (i.e.,  the  process  and  criteria  used  and  who 
identified  them).  Describes  the  scale  used  for  the  assessments  [usually  Technol¬ 
ogy  Readiness  Levels  (TRLs)]. 

States  what  analyses  and  investigations  were  performed  when  making  the 
assessment  (e.g.,  examination  of  test  setups,  discussions  with  test  personnel, 
analysis  of  test  data,  review  of  related  technology,  and  so  forth). 

This  is  only  a  broad  description  of  the  process.  Paragraph  3.3  presents  an 
opportunity  to  include  more  detail. 

3.2  CTEs 

Shows  the  work  breakdown  structure  (WBS)  or  systems  architecture  and  the 
CTEs.  Lists  the  technologies  included  in  the  TRA.  Explains  the  criterion  for  tech¬ 
nologies  that  were  included  on  the  list  of  CTEs.  Describes  the  environment  sur¬ 
rounding  each  CTE.  A  table  that  lists  the  technology  name  and  includes  a  few 
words  that  describe  the  technology,  its  function,  and  the  environment  is  appropri¬ 
ate.  The  names  of  these  CTEs  should  be  used  consistently  throughout  the  remain¬ 
der  of  the  document. 

Any  additional  technology  elements  that  the  Component  S&T  Executive 
considers  critical  should  be  included. 

3.3  Assessment  of  Maturity 

3.3.1  First  CTE  or  Category  of  Technology 

Describes  the  technology  (subsystem,  component,  or  technology).  Describes 
the  function  it  performs  and,  if  needed,  how  it  relates  to  other  parts  of  the  system. 
Provides  a  synopsis  of  development  history  and  status.  This  can  include  facts 
about  related  uses  of  the  same  or  similar  technology,  numbers  or  hours  of  testing 
of  breadboards,  numbers  of  prototypes  built  and  tested,  relevance  of  the  test  con¬ 
ditions,  and  results  achieved. 
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Describes  the  environment  in  which  the  technology  has  been  demonstrated. 
Provides  a  brief  analysis  of  the  similarities  between  the  demonstrated  environ¬ 
ment  and  the  intended  operational  environment. 

Finally,  applies  the  criteria  for  TRLs  and  assigns  a  readiness  level  to  the 
technology.  States  the  readiness  level  (e.g.,  TRL  5)  and  the  rationale  for  choosing 
this  readiness  level. 

Provides  extensive  references  to  papers,  presentations,  data,  and  facts  that 
support  the  assessment.  Includes  data  tables  and  graphs  that  illustrate  that  a  key 
fact  is  appropriate. 

If  the  CTEs  presented  are  in  categories  (e.g.,  airframe  or  sensors),  the 
information  specified  in  the  previous  paragraph  (e.g.,  describing  the  technology, 
describing  the  function  it  performs,  and  so  forth)  should  be  provided  for  each 
CTE  within  a  category. 

3.3.2  Next  CTE  or  Category  of  Technology 

For  the  other  CTEs,  this  paragraph  and  the  following  paragraphs  (e.g.,  3.3.3, 
3.3.4,  and  so  forth)  present  the  same  type  of  information  that  was  presented  in 
paragraph  3.3.1. 

3.4  Summary  of  TRLs  by  Technology 

Presents  a  table  that  lists  the  CTEs  and  presents  the  TRL  assigned,  along 
with  a  short  explanation  (one  sentence  or  a  list  of  factors). 

4.0  Conclusion 

States  the  Component  S&T  Executive’s  position  concerning  the  maturity  of 
the  technologies  and  whether  this  maturity  is  adequate  for  the  system  to  enter  the 
next  stage  of  development.  If  the  position  supports  entering  the  next  stage  even 
though  some  CTEs  are  less  mature  than  would  ordinarily  be  expected,  explains 
what  circumstances  or  planned  work  justifies  this  position.  Includes  references  to 
a  separately  submitted  Technology  Maturation  Plan  (TMP)  for  each  immature 
CTE. 

The  TRA  should  be  signed  “Approved  By”  the  Component  S&T  Executive, 
or  it  should  be  transmitted  with  a  cover  memorandum  that  clearly  states  that  the 
TRA  represents  the  position  of  the  Component  S&T  Executive.  In  effect,  the 
Component  S&T  Executive  must  certify  that  he/she  stands  behind  the  statements 
in  the  conclusion. 

Finally,  the  TRA  should  be  signed  by  the  Component  Acquisition  Executive 
(CAE). 
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SECTION  5. 

GUIDANCE  AND  BEST  PRACTICES  FOR  THE  USE  OF  TMPs 


5.1  INTRODUCTION 

DoD  has  a  long  history  of  accepting  high  technology  risk  and  suffering  the  conse¬ 
quences.  “Excessive  optimism  in  drawing  up  performance  specifications  can  make  the 
development  so  difficult  that  it  must  fail  or  take  much  longer  and  cost  much  more  than 
planned,  or  require  a  downgrading  of  the  requirements.  It  is  not  unusual  for  weapon  sys¬ 
tem  requirements  to  be  so  optimistic  that  several  inventions  or  advances  in  the  state  of  the 
art  are  needed  on  schedule  if  the  development  is  to  succeed.”38  More  than  40  years  after 
this  observation,  the  situation  has  not  changed  substantially.  The  current  defense  acquisi¬ 
tion  system  attempts  to  solve  this  problem  by  requiring  that  technologies  for  a  new  sys¬ 
tem  be  mature  before  being  used  in  Systems  Integration,  the  first  part  of  SDD. 

5.2  THE  PROPER  TIME  TO  MATURE  TECHNOUOGY 

DoD  policy  on  technology  risk  is  quite  clear:  “PMs  shall  reduce  technology  risk, 
demonstrate  technologies  in  a  relevant  environment,  and  identify  technology  alternatives, 
prior  to  program  initiation”  (DoDI  5000.1,  paragraph  El.  14).  Program  initiation  is  nor¬ 
mally  at  Milestone  B. 


The  management  and  mitigation  of  technology  risk,  which  allows  less  costly  and 
less  time-consuming  systems  development,  is  a  crucial  part  of  overall  program 
management  and  is  especially  relevant  to  meeting  cost  and  schedule  goals. 
Objective  assessment  of  technology  maturity  and  risk  shall  be  a  routine  aspect  of 
DoD  acquisition.  Technology  developed  in  S&T  or  procured  from  industry  or 
other  sources  shall  have  been  demonstrated  in  a  relevant  environment  or,  pref¬ 
erably,  in  an  operational  environment  to  be  considered  mature  enough  to  use  for 
product  development  in  systems  integration.  Technology  readiness  assessments 
and,  where  necessary,  independent  assessments,  shall  be  conducted.  If  tech¬ 
nology  is  not  mature,  the  DoD  Component  shall  use  alternative  technology  that  is 
mature  and  that  can  meet  the  user’s  needs.  (DoDI  5000.2,  paragraph  3. 7. 2. 2) 


38  C.J.  Hitch  and  R.N.  McKean,  The  Economics  of  Defense  in  the  Nuclear  Age,  Atheneum  Press,  New 
York,  1965,  cited  in  E.H.  Conrow,  Effective  Risk  Management:  Some  Keys  To  Success  (p.  4), 
American  Institute  of  Aeronautics  and  Astronautics,  Inc.,  Reston,  VA,  2000. 
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All  acquisition  personnel,  especially  PMs,  should  understand  that  acquisition  pro¬ 
grams  are  to  use  mature  technology.  The  overall  acquisition  management  framework  is 
designed  to  provide  time  for  identifying  the  technology  needed  and  time  for  developing 
the  needed  technology.  These  intervals  are  designated  Concept  Refinement  and 
Technology  Development. 

The  TDS  formulated  during 
the  Concept  Refinement  phase 
should  show  how  the  technologies 
(those  known  at  Milestone  A  to  be 
critical  to  the  successful  realization 
of  the  chosen  concept)  will  be 
demonstrated  in  a  relevant  environ¬ 
ment  before  they  are  used  in  Sys¬ 
tem  Development.  Any  exception  to 
this  approach  should  be  brought  to  the  attention  of  the  MDA,  and  this  departure  from 
DoD  policy  should  be  fully  justified  before  asking  the  MDA  to  approve  entry  into  the 
Technology  Development  phase  for  potential  ACAT  ID  and  IAM  programs. 

The  chosen  concept  continues  to  evolve  and  become  better  defined  throughout  the 
Technology  Development  phase.  This  process  can  lead  to  a  different  set  of  preferred 
technologies,  some  of  which  might  be  CTEs.  When  this  occurs,  the  program  should  con¬ 
sider  technology  maturity  in  making  the  technology  choices  and  should  execute  a  pro¬ 
gram  to  mature  all  CTEs  before  reaching  Milestone  B.  DoDI  5000.2  makes  it  clear  that 
programs  should  be  planned  so  that  System  Development  can  use  only  mature 
technologies. 

The  project  shall  exit  Technology  Development  when  an  affordable  incre¬ 
ment  of  militarily  useful  capability  has  been  identified,  the  technology  for 
that  increment  has  been  demonstrated  in  a  relevant  environment,  and  a 
system  can  be  developed  for  production  within  a  short  time  frame  (nor¬ 
mally  less  than  5  years);  or  when  the  MDA  decides  to  terminate  the  effort. 

...  A  Milestone  B  decision  follows  the  completion  of  Technology  Devel¬ 
opment.”  (DoDI  5000.2,  paragraph  3.6.7). 


Best  Practice 

For  ACAT  ID  and  IAM  programs,  the 
sponsoring  Component  should  ensure 
that  the  TDS  (at  Milestone  A) 
includes  activities,  schedule,  and 
funding  to  demonstrate  the  identified 
CTEs  of  the  chosen  concept  in  a 
relevant  environment. 
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5.3  TMPs  AT  MILESTONE  B39 


DoD  policy  for  technology 
maturation  at  Milestone  B  is 
unambiguous.  Either  schedule 
Milestone  B  for  a  date  after  the 
CTEs  will  be  mature  or  use  other 
technologies  that  are  already 
mature.  Nevertheless,  PMs  have  taken  programs  to  Milestone  B  with  immature  CTEs 
(i.e.,  not  demonstrated  in  the  relevant  environment).  In  this  case,  the  MDA  decision 
options  include  rescheduling  Milestone  B,  directing  the  use  of  an  alternative  technology 
(perhaps  only  in  the  initial  increment),  or  conditionally  approving  entry  into  SDD. 

Conditional  approval  to  enter  SDD  implies  that  certain  critical  conditions  have 
been  met,  namely: 

•  A  sound  technical  basis  exists  for  expecting  the  immature  technology  to 
prove  adequate  after  a  demonstration. 

•  A  substitute  mature  technology  exists,  and  this  technology  can  be  used  in 
case  the  demonstration  is  unsuccessful. 

•  The  program  plan  can  accommodate  use  of  either  technology  from  funding, 
performance,  and  schedule  perspectives. 

According  to  the  Defense  Acquisition  Guidebook  these  conditions  must  be  cap¬ 
tured  in  a  TMP.  “If  the  system  does  not  meet  pre-defined  Technology  Readiness  Level 
scores,  then  a  Critical  Technology  Element  maturation  plan  is  identified.  This  plan 
explains  in  detail  how  the  Technology  Readiness  Level  will  be  reached  prior  to  the  next 
milestone  decision  date  or  relevant  decision  point”  (Section  4.3. 2.4.3). 

The  TMP  is  different  from  the  TDS  that  was  prepared  before  Milestone  A.  The 
TMP  should  be  more  specific  regarding  what  is  going  to  be  done,  what  test  articles  will 
be  made,  the  TRL  levels  that  will  be  achieved,  the  schedule  of  demonstrations,  the  fall¬ 
back  technology  and  its  maturity,  and  the  decision  date  for  choosing  between  the  pre¬ 
ferred  technology  and  the  fail-back  technology.  Section  5.5.1  of  this  document  provides 
an  outline  for  a  TMP. 


39  In  an  evolutionary  acquisition  program,  a  separate  Technology  Development  phase  and  Milestone  B 
are  required  for  each  increment.  The  reason  for  a  new  increment  might  be  to  introduce  new  technology 
or  to  extend  the  operating  environment,  either  of  which  could  result  in  the  use  of  immature  technology. 
Consequently,  each  increment  should  be  examined  individually. 


Best  Practice 

All  CTEs  should  be  identified  and 
successfully  demonstrated  in  a  rele¬ 
vant  environment  (a  TRL  6  or  higher) 
before  Milestone  B. 
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The  TMP  is  a  commitment  by  the  PM  and  the  Component.  It  should  be  approved 
or  certified  by  the  CAE  and  included  with  the  TRA  for  ACAT  ID  and  IAM  programs  to 
give  the  MDA  a  basis  for  the  Milestone  B  decision. 

When  the  MDA  authorizes 
a  program  to  enter  SDD  with 
immature  technology,  the  Acqui¬ 
sition  Decision  Memorandum 
(ADM)  usually  calls  for  greater 
than  normal  oversight  by  the  OSD 
staff.  It  also  sometimes  requires 
additional  reporting— possibly  even 
a  new  TRA— to  be  provided  by  a 
specified  time  during  System  Integration. 

5.4  TMPs  AT  MILESTONE  C 

A  TRA  is  also  required  before  approval  to  enter  LRIP  at  Milestone  C  for  hard¬ 
ware  systems  or  limited  deployment  in  support  of  operational  testing  for  MAIS  programs 
or  software-intensive  systems  that  have  no  production  components.  Just  as  TRL  6  is  usu¬ 
ally  required  for  Milestone  B,  TRL  7  is  usually  required  for  Milestone  C.  By  Mile¬ 
stone  C,  prototypes  have  been  tested  in  field  trials  under  conditions  that  simulate  the 
operational  environment.  Success  in  these  tests  can  usually  be  taken  as  demonstration  at 
TRL  7  for  the  performance  technologies.40  However,  unplanned  technologies  could  have 
been  incorporated  in  the  design  during  SDD,  so  a  careful  review  of  the  design  should  be 
conducted  to  find  any  such  technologies  and  then  to  assess  their  maturity.  Under  no  cir¬ 
cumstances  should  a  critical  performance  technology  be  less  than  TRL  7  at  Milestone  C. 

The  TRA  at  Milestone  C  is  most  important  for  manufacturing  technologies.  Proto¬ 
types  are  often  built  by  methods  that  are  not  suitable  for  production,  so  the  testing  of  pro¬ 
totypes  does  not  usually  tell  much  about  the  maturity  of  the  manufacturing  technologies 
that  must  be  used  to  achieve  the  production  rate,  production  cost,  and  low  defect  rate  that 
are  needed. 


Best  Practice 

If  some  overriding  circumstance  causes 
the  program  to  have  a  Milestone  B 
review  before  all  CTEs  are  at  TRL  6 
or  above,  include,  as  part  of  the  TRA, 
a  TMP  for  each  deficient  technology 
and  a  rationale  for  proceeding  to  SDD 
while  maturation  continues. 


40  “Performance  technologies"  influence  the  performance  of  the  produced  system,  as  distinguished  from 
“manufacturing  technologies,"  which  influence  construction  of  that  system. 
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At  TRL  7,  the  maturity  of  a  manufacturing  technology  should  be  as  follows: 

•  Manufacturing  processes,  materials  and  assembly  methods  have  been  devel¬ 
oped  for  a  production  environment— ideally  in  a  pre-production  facility  or 
better. 

•  The  design  is  maturing,  key  materials  and  process  characteristics  have  been 
identified,  and  planning  is  taking  place  for  managing  process  controls,  as 
appropriate. 

•  A  detailed  manufac¬ 
turing  risk  assessment 
has  been  performed. 

This  assessment  cov¬ 
ers  industrial  base 
infrastructure  (facili¬ 
ties  and  manpower), 
materials  (availability, 
producibility  charac¬ 
teristics),  methods 
(mature  processes),  measurement  (inspection  and  test  equipment),  and  costs. 

•  A  quality  management  structure  has  been  identified. 

•  Initial  goals  have  been  set  for  yields,  quality,  and  reliability. 

FRP  (post  LRIP)  should  not  be  initiated  if  a  critical  manufacturing  technology  has 
not  reached  TRL  8  — successfully  qualified  through  test  and  demonstration— or  TRL  9. 
This  implies  the  following: 

•  Manufacturing  processes,  materials,  and  assembly  methods  demonstrated  on 
production-representative  articles  with  no  known  significant  manufacturing 
risk 

•  Yields,  quality,  and  reliability  within  25  percent  of  goals 

•  Design  mature  (process  requirements  proven  and  validated) 

•  Quality  management  structures  in  place. 

5.5  PREPARING  TMPs 

Figure  5-1  shows  a  possible  schedule  of  activities  to  mitigate  technology  risk. 
While  this  is  a  continuation  of  the  schedules  shown  in  Section  3,  the  time  scaling  is  dif¬ 
ferent.  TMP  oversight  is  a  continuous  activity  over  a  long  period  of  time.  Arbitrary  loca¬ 
tions  have  been  used  for  the  three  technical  reviews  that  occur  during  Systems 


Best  Practice 

If  the  critical  manufacturing  technolo¬ 
gies  have  not  been  successfully  qualified 
through  test  and  demonstration  (TRL  8) 
by  Milestone  C,  include  with  the  Mile¬ 
stone  C  TRA  a  TMP  for  the  immature 
manufacturing  technologies. 


5-5 


SFR 

k 


POR 


CDR 


TMP  Development 
TMP  Coordination 
Milestone  Review 
TMP  Oversight 


Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week  Week 
14  13  12  11  10  9  8  7  6  5  4  3  2  1 


k  ik 


Figure  5-1.  Representative  Schedule  for  TMP  Preparation  and  Oversight 

Integration:  the  System  Function  Review  (SFR),  the  Preliminary  Design  Review  (PDR), 
and  the  Critical  Design  Review  (CDR).  The  following  subsections  describe  the  template, 
activities,  and  key  player  roles  and  responsibilities. 


5.5.1  TMP  Development 

The  following  outline  for  a  TMP  includes  the  most  essential  items: 

•  Title 

•  Statement  of  the  problem 

-  Describe  the  technology  and  its  maturity  status 

-  Say  how  this  technology  would  be  used  in  the  system 

•  Solution  options 

-  Benefits  of  using  the  preferred  technology 

-  Fall-back  options  and  the  consequences  of  each  option 

•  Maturation  program  plan  with  schedule 

-  Describe  key  activities  for  the  preferred  technology 

-  Describe  preparations  for  using  an  alternative  technology 

-  Show  the  latest  time  that  an  alternative  technology  can  be  chosen 

•  Specific  actions  to  be  taken  (what  will  be  done  and  by  whom) 

-  What  prototypes  or  EMDs  will  be  built? 

-  What  tests  will  be  run? 

-  How  does  the  test  environment  relate  to  the  operational  environment? 

-  What  threshold  performance  must  be  met? 

-  What  TRL  will  be  achieved? 

•  Status  of  funding  to  perform  this  technology  maturation. 
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Key  Player  Roles  and  Responsibilities  in  TMP  Development 

•  PM.  The  PM  is  responsible  for  developing  the  TMP.  When  a  PM  anticipates 
that  a  CTE  will  be  less  mature  than  TRL  6  at  Milestone  B  or  a  critical 
manufacturing  technology  is  less  than  TRL  8  at  Milestone  C,  prepare  a  TMP 
and  submit  that  plan  through  the  Component  S&T  Executive  to  be  included 
with  the  Component  TRA. 

•  Component  S&T  Executive.  Assign  an  AO  to  participate  in  the  process  to 
the  extent  that  his/her  participation  is  considered  useful  and  valuable. 

5.5.2  TMP  Coordination 

The  TMP  coordination  should  be  concurrent  with  the  TRA  coordination. 

Key  Player  Roles  and  Responsibilities  in  TMP  Coordination 

•  PM.  Submit  the  plan  to  the  Component  S&T  Executive.  An  information 
copy  should  be  sent  to  the  DUSD(S&T)  for  ACAT  ID  and  ACAT  IAM  pro¬ 
grams.  Provide  any  additional  information  requested  by  the  Component  S&T 
Executive  or  the  DUSD(S&T). 

•  Component  S&T  Executive.  Review  the  plan  and  coordinate  with  the  PM 
on  any  additions  or  deletions  and  on  any  additional  information  needed.  For 
ACAT  ID  and  ACAT  IAM  programs,  approve  the  TMP,  accept  responsibil¬ 
ity  for  its  accuracy,  submit  the  plan  (e.g.,  sign  a  forwarding  memorandum)  to 
the  CAE  or  Agency  Head  and,  at  the  same  time,  send  an  information  copy  to 
the  DUSD(S&T). 

•  CAE  or  Agency  Head.  For  ACAT  ID  and  ACAT  IAM  programs,  certifies 
that  the  plan  is  a  commitment  of  the  Component  and  submits  the  plan  to  the 
DUSD(S&T). 

5.5.3  TMP  Oversight 

Technical  reviews  provide  the  PM  an  integrated  technical  assessment  of  program 
technical  risk  and  the  readiness  to  proceed  to  the  next  technical  phase.  These  technical 
reviews  are  conducted  between  the  Program  Management  Office  (PMO)  and  the  system 
developer. 

The  Defense  Acquisition  Guidebook  identifies  three  such  reviews  during  System 
Integration:  SFR,  PDR,  and  CDR.  These  reviews  provide  opportunities  to  assess  the  pro¬ 
gress  of  the  technology  maturation  activities  and  to  inform  OSD  of  the  results.  Close 
attention  to  the  maturation  progress  is  important  because  any  delay  or  failure  could  jeop¬ 
ardize  a  program’s  schedule. 
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Key  Player  Roles  and  Responsibilities  in  TMP  Oversight 

•  PM.  Notify  the  DUSD(S&T)  and  the  Component  S&T  Executive  whenever  a 
decision  is  made  to  use  a  mature,  alternative  technology  or  whenever  any 
deviation  from  the  schedule  in  the  TMP  occurs.  Report  the  status  and  results 
at  OIPT  meetings. 

•  Component  S&T  Executive.  Monitor  all  technology  maturation  efforts  and 
demonstration  activities  included  in  the  plan.  Support  the  maturation  efforts 
as  required  by  the  plan. 

•  DUSD(S&T).  Monitor  the  technology  maturation  status  at  technical  reviews 
and  OIPT  meetings. 
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GLOSSARY 


ACAT 

Acquisition  Category 

ADM 

Acquisition  Decision  Memorandum 

AIS 

Automated  Information  System 

AKSS 

AT&L  Knowledge  Sharing  System 

AO 

Action  Officer 

AoA 

Analysis  of  Alternatives 

APB 

Acquisition  Program  Baseline 

ASD(C3I) 

Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications,  and  Intelligence 

ASD(NII) 

Assistant  Secretary  of  Defense  for  Networks  and 
Information  Integration 

CAE 

Component  Acquisition  Executive 

CDD 

Capability  Development  Document 

CDR 

Critical  Design  Review 

CIO 

Chief  Information  Officer 

CJCSI 

Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 

CJCSM 

Chairman  of  the  Joint  Chiefs  of  Staff  Manual 

COTS 

commercial-off-the-shelf 

CPD 

Capability  Production  Document 

CS&T 

Component  Science  and  Technology 

CTE 

Critical  Technology  Element 

DAB 

Defense  Acquisition  Board 

DARC 

Defense  Acquisition  Resource  Center 

DAU 

Defense  Acquisition  University 

DDR&E 

Director  of  Defense  Research  and  Engineering 

DoD 

Department  of  Defense 

DoDD 

DoD  Directive 

DoDI 

DoD  Instruction 

GL-1 


DOTMLPF 

DRR 

DT&E 

DTC 

DUSD(S&T) 

EDM 

FOC 

FRP 

GAO 

GOTS 

ICD 

IOC 

IOT&E 

IT  OIPT 

IT 

ITA 

ITAB 

JCIDS 

JDMTP 

JFCOM 

JROC 

KPP 

LRIP 

M&S 

MAIS 

MDA 

MDAP 

MRL 

MS 

NASA 


doctrine,  organization,  training,  materiel,  leadership  and 
education,  personnel,  and  facilities 

Design  Readiness  Review 

development,  test,  and  evaluation 

design-to-cost 

Deputy  Under  Secretary  of  Defense  for  Science  and 
Technology 

Engineering  Development  Model 

full  operational  capability 

Full  Rate  Production 

government  Accountability  Office 

Government-off-the-shelf 

Initial  Capabilities  Document 

initial  operational  capability 

Initial  Operational  Test  and  Evaluation 

Information  Technology  Overarching  Integrated  Product 
Team 

Information  Technology 

Independent  Technical  Assessment 

Information  Technology  Acquisition  Board 

Joint  Capabilities  Integration  and  Development  System 

Joint  Defense  Manufacturing  Technology  Panel 

Joint  Forces  Command 

Joint  Requirements  Oversight  Council 

key  performance  parameter 

Low  Rate  Initial  Production 

modeling  and  simulation 

Major  Automated  Information  System 

Milestone  Decision  Authority 

Major  Defense  Acquisition  Program 

Manufacturing  Readiness  Level 

Milestone 

National  Aeronautics  and  Space  Administration 
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OIPT 

Overarching  Integrated  Product  Team 

OSD 

Office  of  the  Secretary  of  Defense 

OT&E 

Operational  Test  and  Evaluation 

PDR 

Preliminary  Design  Review 

PEO 

Program  Executive  Office 

PM 

Program  Manager 

PMO 

Program  Management  Office 

R&D 

research  and  development 

RDT&E 

research,  development,  test,  and  evaluation 

S&T 

Science  and  Technology 

SAIC 

Science  Applications  International  Corporation 

SDD 

System  Development  and  Demonstration 

SFR 

System  Function  Review 

Sim/Stim 

Simulation/Stimulation 

SPO 

System  Program  Office 

SRR 

Systems  Requirements  Review 

TDS 

Technology  Development  Strategy 

TEAO 

Test,  Evaluation,  Analysis,  and  Operation 

TMP 

Technology  Maturation  Plan 

TRA 

Technology  Readiness  Assessment 

TRL 

Technology  Readiness  Level 

TTA 

Technology  Transition  Agreement 

USD(AT&L) 

Under  Secretary  of  Defense  for  Acquisition,  Technology, 
and  Logistics 

WBS 

work  breakdown  structure 
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The  DoD  documents  relevant  to  TRAs  are 


•  Department  of  Defense  Directive  (DoDD)  5000.1,  The  Defense  Acquisition 
System,  dated  May  12,  2003 

•  Department  of  Defense  Instruction  (DoDI)  5000.2,  Operation  of  the  Defense 
Acquisition  System,  dated  May  12,  2003 

•  Defense  Acquisition  Guidebook. 

For  background  and  reference,  the  portions  of  these  documents  relevant  to  tech¬ 
nology  readiness  are  shown  below.  These  documents  appear  on  Internet  Web  site 
http://akss.dau.mil/darc/darc.html  in  their  entirety. 

A.l  EXTRACTS  FROM  DoDD  5000.1,  DATED  MAY  12,  2003 

A.  1 . 1  Section  4.  Policy 

•  4.3.  The  following  policies  shall  govern  the  Defense  Acquisition  System: 

4.3.1.  Flexibility.  There  is  no  one  best  way  to  structure  an  acquisition  pro¬ 
gram  to  accomplish  the  objective  of  the  Defense  Acquisition  System.  MDAs 
and  PMs  shall  tailor  program  strategies  and  oversight,  including  documenta¬ 
tion  of  program  information,  acquisition  phases,  the  timing  and  scope  of 
decision  reviews,  and  decision  levels,  to  fit  the  particular  conditions  of  that 
program,  consistent  with  applicable  laws  and  regulations  and  the  time-sensi¬ 
tivity  of  the  capability  need. 

4.3.2.  Responsiveness.  Advanced  technology  shall  be  integrated  into  produc¬ 
ible  systems  and  deployed  in  the  shortest  time  practicable.  Approved,  time- 
phased  capability  needs  matched  with  available  technology  and  resources 
enable  evolutionary  acquisition  strategies.  Evolutionary  acquisition  strategies 
are  the  preferred  approach  to  satisfying  operational  needs.  Spiral  develop¬ 
ment  is  the  preferred  process  for  executing  such  strategies. 

A.  1.2  Enclosures 

•  Enclosure  1:  Additional  Policy 

El.  14.  Knowledge-Based  Acquisition.  PMs  shall  provide  knowledge  about 
key  aspects  of  a  system  at  key  points  in  the  acquisition  process.  PMs  shall 
reduce  technology  risk,  demonstrate  technologies  in  a  relevant  environment, 
and  identify  technology  alternatives,  prior  to  program  initiation.  They  shall 
reduce  integration  risk  and  demonstrate  product  design  prior  to  the  design 
readiness  review.  They  shall  reduce  manufacturing  risk  and  demonstrate  pro- 
ducibility  prior  to  full-rate  production. 
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E1.28.  Technology  Development  and  Transition.  The  Science  and  Tech¬ 
nology  (S&T)  program  shall: 

E.l.28.1.  Address  user  needs; 

E.l.28.2.  Maintain  a  broad-based  program  spanning  all  Defense-relevant  sci¬ 
ences  and  technologies  to  anticipate  future  needs  and  those  not  being  pursued 
by  civil  or  commercial  communities; 

El.28.3.  Preserve  long-range  research;  and 

E.l.28.4.  Enable  rapid,  successful  transition  from  the  S&T  base  to  useful 
military  products. 

A.2  EXTRACTS  FROM  DoDI  5000.2,  DATED  MAY  12,  2003 

A.2.1  Section  2.  Applicability  and  Scope 

•  2.2.  All  defense  technology  projects  and  acquisition  programs.  Some  require¬ 
ments,  where  stated,  apply  only  to  Major  Defense  Acquisition  Programs 
(MDAPs)  and  Major  Automated  Information  System  (MAIS)  programs. 

A.2.2  Section  3.  Procedures 

•  3.4.  User  Needs  and  Technology  Opportunities 

3.4.1.  The  capability  needs  and  acquisition  management  systems  shall  use 
Joint  Concepts,  integrated  architectures,  and  an  analysis  of  doctrine,  organi¬ 
zation,  training,  materiel,  leadership,  personnel,  and  facilities  (DOTMLPF)  in 
an  integrated,  collaborative  process  to  define  desired  capabilities  to  guide  the 
development  of  affordable  systems.  The  Chairman  of  the  Joint  Chiefs  of 
Staff,  with  the  assistance  of  the  Joint  Requirements  Oversight  Council,  shall 
assess  and  provide  advice  regarding  military  capability  needs  for  defense 
acquisition  programs.  The  process  through  which  the  Chairman  provides  his 
advice  is  described  in  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 
3170.01  (reference  (g)).  Representatives  from  multiple  DoD  communities 
shall  assist  in  formulating  broad,  time-phased,  operational  goals,  and  des¬ 
cribing  requisite  capabilities  in  the  Initial  Capabilities  Document  (ICD).  They 
shall  examine  multiple  concepts  and  materiel  approaches  to  optimize  the  way 
the  Department  of  Defense  provides  these  capabilities.  The  examination  shall 
include  robust  analyses  that  consider  affordability,  technology  maturity,  and 
responsiveness. 

•  3.5.  Concept  Refinement 

3.5.2.  Concept  Refinement  begins  with  the  Concept  Decision.  The  MDA  des¬ 
ignates  the  lead  DoD  Component(s)  to  refine  the  initial  concept  selected, 
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approves  the  AoA  plan,  and  establishes  a  date  for  a  Milestone  A  review.  The 
MDA  decisions  shall  be  documented  in  an  Acquisition  Decision  Memoran¬ 
dum  (ADM).  This  effort  shall  normally  be  funded  only  for  the  concept 
refinement  work.  The  MDA  decision  to  begin  Concept  Refinement  DOES 
NOT  mean  that  a  new  acquisition  program  has  been  initiated.  The  tables  in 
enclosure  3  identify  all  statutory  and  regulatory  requirements  for  the  Concept 
Refinement  decision. 

3.5.3.  The  ICD  and  the  AoA  plan  shall  guide  Concept  Refinement.  The  focus 
of  the  AoA  is  to  refine  the  selected  concept  documented  in  the  approved 
ICD.  The  AoA  shall  assess  the  critical  technologies  associated  with  these 
concepts,  including  technology  maturity,  technical  risk,  and,  if  necessary, 
technology  maturation  and  demonstration  needs.  To  achieve  the  best  possible 
system  solution,  emphasis  shall  be  placed  on  innovation  and  competition. 
Existing  commercial-off-the-shelf  (COTS)  functionality  and  solutions  drawn 
from  a  diversified  range  of  large  and  small  businesses  shall  be  considered. 

•  3.6.  Technology  Development 

3.6.1.  Purpose.  The  purpose  of  this  phase  is  to  reduce  technology  risk  and  to 
determine  the  appropriate  set  of  technologies  to  be  integrated  into  a  full  sys¬ 
tem.  Technology  Development  is  a  continuous  technology  discovery  and 
development  process  reflecting  close  collaboration  between  the  S&T  com¬ 
munity,  the  user,  and  the  system  developer.  It  is  an  iterative  process  designed 
to  assess  the  viability  of  technologies  while  simultaneously  refining  user 
requirements. 

3.6.2.  The  project  shall  enter  Technology  Development  at  Milestone  A  when 
the  MDA  has  approved  the  TDS.  The  tables  in  enclosure  3  identify  all  statu¬ 
tory  and  regulatory  requirements  applicable  to  Milestone  A.  This  effort  nor¬ 
mally  shall  be  funded  only  for  the  advanced  development  work.  For  business 
area  capabilities,  commercially  available  solutions  shall  be  employed.  (A 
toolkit  of  best  practices  is  available  at  http://deskbook.dau.mil).  A  favorable 
Milestone  A  decision  DOES  NOT  mean  that  a  new  acquisition  program  has 
been  initiated. 

3.6.3.  Shipbuilding  programs  may  be  initiated  at  the  beginning  of  Technol¬ 
ogy  Development.  The  information  required  in  the  tables  at  enclosure  3  shall 
support  program  initiation.  A  cost  assessment  shall  be  prepared  in  lieu  of  an 
independent  cost  estimate  (ICE),  and  a  preliminary  assessment  of  the  matur¬ 
ity  of  key  technologies  shall  be  provided. 
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•  3.7.  System  Development  and  Demonstration 

3.7.1.  Purpose 

3.7.1.2.  SDD  has  two  major  efforts:  System  Integration  and  System  Dem¬ 
onstration.  The  entrance  point  is  Milestone  B,  which  is  also  the  initiation  of 
an  acquisition  program.  There  shall  be  only  one  Milestone  B  per  program  or 
evolutionary  increment.  Each  increment  of  an  evolutionary  acquisition  shall 
have  its  own  Milestone  B.  The  tables  in  enclosure  3  identify  the  statutory  and 
regulatory  requirements  that  shall  be  met  at  Milestone  B.  For  Shipbuilding 
Programs,  the  required  program  information  shall  be  updated  in  support  of 
the  Milestone  B  decision,  and  the  ICE  shall  be  completed.  The  lead  ship  in  a 
class  shall  normally  be  authorized  at  Milestone  B.  Technology  readiness 
assessments  shall  consider  the  risk  associated  with  critical  subsystems  prior 
to  ship  installation.  Long  lead  for  follow  ships  may  be  initially  authorized  at 
Milestone  B,  with  final  authorization  and  follow  ship  approval  by  the  MDA 
dependent  on  completion  of  critical  subsystem  demonstration  and  an  updated 
assessment  of  technology  maturity. 

3.7.2.  Entrance  Criteria.  Entrance  into  this  phase  depends  on  technology 
maturity  (including  software),  approved  requirements,  and  funding.  Unless 
some  other  factor  is  overriding  in  its  impact,  the  maturity  of  the  technology 
shall  determine  the  path  to  be  followed.  Programs  that  enter  the  acquisition 
process  at  Milestone  B  shall  have  an  ICD  that  provides  the  context  in  which 
the  capability  was  determined  and  approved,  and  a  CDD  that  describes  spe¬ 
cific  program  requirements. 

3.7.2.2.  The  management  and  mitigation  of  technology  risk,  which  allows 
less  costly  and  less  time-consuming  systems  development,  is  a  crucial  part  of 
overall  program  management  and  is  especially  relevant  to  meeting  cost  and 
schedule  goals.  Objective  assessment  of  technology  maturity  and  risk  shall 
be  a  routine  aspect  of  DoD  acquisition.  Technology  developed  in  S&T  or 
procured  from  industry  or  other  sources  shall  have  been  demonstrated  in  a 
relevant  environment  or,  preferably,  in  an  operational  environment  to  be  con¬ 
sidered  mature  enough  to  use  for  product  development  in  systems  integra¬ 
tion.  Technology  readiness  assessments,  and  where  necessary,  independent 
assessments,  shall  be  conducted.  If  technology  is  not  mature,  the  DoD  Com¬ 
ponent  shall  use  alternative  technology  that  is  mature  and  that  can  meet  the 
user’s  needs. 
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A.2.3  Enclosures 

•  Enclosure  3:  Statutory,  Regulatory,  and  Contract  Reporting  Informa¬ 
tion  and  Milestone  Requirements 

E.3.1.  Tables  E3.T1,  E3.T2,  and  E3.T3,41  below,  show  the  information 
requirements  for  all  milestones  and  phases,  both  statutory  and  regulatory,  to 
include  contract  reporting.  MDAs  may  tailor  regulatory  program  information 
to  fit  the  particular  conditions  of  an  individual  program.  A  non-mandatory 
guidebook  shall  support  this  Instruction  to  provide  best  practices,  lessons 
learned,  and  expectations  for  the  information  required  by  these  tables.  Issues 
regarding  the  intent  of  the  expectations  described  in  the  guidebook  shall  be 
resolved  by  the  MDA.  The  AT&L  Knowledge  Sharing  System  (formerly 
Defense  Acquisition  Deskbook)  contains  a  library  of  mandatory  policy  and 
regulations  and  discretionary  practices  and  advice.  The  Internet  Web  site 
address  is  http://deskbook.dau.mik. 


Table  E3.T1.  Statutory  Information  Requirements 


Information  Required 

Applicable  Statute 

When  Required 

The  following  information  requirements  are  statutory  for  both  MDAPs  anc 

MAIS  acquisition  programs 

Consideration  of  Technology  Issues 

10  U.S.C.  2364,  reference  (q) 

Milestone  (MS)  A 

MS  B 

MS  C 

The  following  information  requirements  are  statutory  for  MDAPs  and  are  applicable  to  MAIS  acquisition  pro¬ 
grams  by  this  Instruction 

Technology  Development  Strategy 

Sec.  803,  Pub.L.  107-314,  refer- 

MSA 

(TDS) 

ence  (an) 

MS  B 

MS  C 

Table  E3.T2.  Regulatory  Information  Requirements 


Information  Required 

Source 

When  Required 

Technology  Readiness  Assessment 

This  Instruction 

Program  Initiation  for  Ships  (prelimi¬ 
nary  assessment) 

MS  B 

MS  C 

Independent  Technology  Assessment 

This  Instruction 

MS  B 

(ACAT  ID  only) 

(if  required  by  DUSD(S&T)) 

MS  C 

Command,  Control,  Communications, 

DoD  Instruction  4630.8  and 

Program  Initiation  for  Ships 

Computers,  and  Intelligence  Support 

DoD  Directive  4630.5, 

MS  B 

Plan  (C4ISP)  (also  summarized  in  the 
acquisition  strategy) 

references  (ar)  and  (as) 

MS  C 

41  The  parts  of  Tables  E3.T1  and  E3.T2  relevant  to  this  discussion  are  included.  Table  E3.T3  is  not 
included  in  this  appendix. 
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A.3  EXTRACTS  FROM  THE  DEFENSE  ACQUISITION  GUIDEBOOK 

A.3.1  Chapter  4.  Systems  Engineering 

•  4.3.  Systems  Engineering  Activities  in  the  System  Life  Cycle 

4.3.2.  Technology  Development  Phase 

4.3.2.5.  Outputs  of  the  Systems  Engineering  Processes  in  Technology 
Development 

-  Preliminary  System  Performance  Specification; 

-  Live-Fire  T&E  Waiver  request; 

-  Test  and  Evaluation  Master  Plan; 

-  Systems  Engineering  Plan; 

-  Programmatic  Environment,  Safety,  and  Occupational  Health  Evaluation 
(PESHE); 

-  NEPA  Compliance  Schedule  (as  required); 

-  Program  Protection  Plan; 

-  Technology  Readiness  Assessment; 

-  Validated  System  Support  and  Maintenance  Objectives  and 
Requirements; 

-  Footprint  Reduction; 

-  Inputs  to  Integrated  Baseline  Review; 

-  Inputs  to  the  Information  Support  Plan; 

-  Inputs  to  the  System  Threat  Assessment; 

-  Inputs  to  the  Capability  Development  Document; 

-  Inputs  to  the  Acquisition  Strategy; 

-  Inputs  to  the  Affordability  Assessment; 

-  Inputs  to  the  Cost  and  Manpower  Estimate;  and 

-  Review  and  update  Designated  Science  and  Technology  Information,  the 
Security  Classification  Guide,  and  the  Counterintelligence  Support  Plan. 

4.3.2.4.3.  Technology  Readiness  Assessment  (TRA) 

Per  DoD  Instruction  5000.2,  the  TRA  is  a  regulatory  information  requirement 
for  all  acquisition  programs.  The  TRA  is  a  systematic,  metrics-based  process 
that  assesses  the  maturity  of  Critical  Technology  Elements.  The  TRA  should 
be  conducted  concurrently  with  other  Technical  Reviews,  specifically  the 


A- 8 


Alternative  Systems  Review,  System  Requirements  Review,  or  the  Produc¬ 
tion  Readiness  Review.  If  a  platform  or  system  depends  on  specific  tech¬ 
nologies  to  meet  system  operational  threshold  requirements  in  development, 
production,  and  operation,  and  if  the  technology  or  its  application  is  either 
new  or  novel,  then  that  technology  is  considered  a  Critical  Technology  Ele¬ 
ment.  The  TRA  should  not  be  considered  a  risk  assessment,  but  it  should  be 
viewed  as  a  tool  for  assessing  program  risk  and  the  adequacy  of  technology 
maturation  planning.  The  TRA  scores  the  current  readiness  level  of  selected 
system  elements,  using  defined  Technology  Readiness  Levels.  The  TRA 
highlights  critical  technologies  and  other  potential  technology  risk  areas  that 
require  program  manager  attention.  The  TRA  essentially  “draws  a  line  in  the 
sand”  on  the  day  of  the  event  for  making  an  assessment  of  technology  readi¬ 
ness  for  critical  technologies  integrated  at  some  elemental  level.  If  the  system 
does  not  meet  pre-defined  Technology  Readiness  Level  scores,  then  a  Criti¬ 
cal  Technology  Element  maturation  plan  is  identified.  This  plan  explains  in 
detail  how  the  Technology  Readiness  Level  will  be  reached  prior  to  the  next 
milestone  decision  date  or  relevant  decision  point.  Completion  of  the  TRA 
should  provide: 

(1)  A  comprehensive  review,  using  an  established  program  Work  Break¬ 
down  Structure  as  an  outline,  of  the  entire  platform  or  system.  This 
review,  using  a  conceptual  or  established  baseline  design  configuration, 
identifies  program  Critical  Technology  Elements; 

(2)  An  objective  scoring  of  the  level  of  technological  maturity  for  each 
Critical  Technology  Element  by  subject  matter  experts; 

(3)  Maturation  plans  for  achieving  an  acceptable  maturity  roadmap  for  Criti¬ 
cal  Technology  Elements  prior  to  critical  milestone  decision  dates;  and 

(4)  A  final  report  documenting  the  findings  of  the  assessment  panel. 

After  the  final  report  is  written,  the  chairman  submits  the  report  to  the  appro¬ 
priate  Service  officials  and  the  program  manager.  Once  approved,  the  report 
and  cover  letter  are  forwarded  to  the  service  acquisition  official.  For  Acqui¬ 
sition  Category  ID  or  IAM  programs,  the  service  acquisition  official  provides 
a  recommendation  to  DDR&E  for  DUSD(S&T)  final  approval.  If  deemed 
necessary,  the  DDR&E  can  conduct  an  Independent  Technical  Assessment 
(IT A)  in  addition  to,  and  totally  separate  from,  the  program  TRA. 

4.3.3.  System  Development  and  Demonstration  Phase 

4.3.3.9.4.  Technology  Readiness  Assessment  (TRA) 

The  program  manager  should  normally  conduct  a  second  TRA  prior  to  Mile¬ 
stone  C.  The  TRA  may  be  held  concurrently  with  other  technical  reviews, 
specifically  System  Requirements  Review,  Critical  Design  Review,  System 
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Verification  Review,  or  Production  Readiness  Review.  Completion  of  this 
TRA  should  provide: 

(1)  An  evaluation  of  system  technology  maturity  based  on  the  Work  Break¬ 
down  Structure; 

(2)  An  objective  scoring  of  the  level  of  technological  maturity;  and 

(3)  Mitigation  plans  for  achieving  acceptable  maturity  prior  to  milestone 
decision  dates. 

4.3.3.10.  Outputs  of  the  Systems  Engineering  Processes  in  System  Devel¬ 
opment  and  Demonstration 

-  Initial  Product  Baseline; 

-  Test  Reports; 

-  Test  and  Evaluation  Master  Plan; 

-  Elements  of  Product  Support; 

-  Systems  Engineering  Plan; 

-  Technology  Readiness  Assessment; 

-  Programmatic  Environment,  Safety,  and  Occupational  Health 
Evaluation; 

-  Inputs  to  the  Capability  Production  Document; 

-  Inputs  to  System  Threat  Assessment; 

-  Inputs  to  the  Information  Support  Plan; 

-  Inputs  to  Cost  and  Manpower  Estimate;  and 

-  Review  and  update  Designated  Science  and  Technology  Information,  the 
Security  Classification  Guide,  and  the  Counterintelligence  Support  Plan. 

A.3.2  Chapter  10.  Decisions,  Assessments,  and  Periodic  Reporting 

•  10.3.  Role  of  Integrated  Product  Teams  (IPTs) 

10.3.1.  Overarching  IPT  (OIPT)  Procedures  and  Assessment 

For  Acquisition  Category  ID  decision  points,  the  OIPT  leader  will  provide 
the  Defense  Acquisition  Board  chair,  co-chair,  principals,  and  advisors  with 
an  integrated  assessment  using  information  gathered  through  the  IPPD  pro¬ 
cess.  The  OIPT  assessment  should  focus  on  core  acquisition  management 
issues  and  should  consider  independent  assessments,  including  technology 
readiness  assessments,  which  the  OIPT  members  normally  prepare.  These 
assessments  typically  occur  in  context  of  the  OIPT  review,  and  should  be 
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reflected  in  the  OIPT  leader’s  report.  There  should  be  no  surprises  at  this 
point-all  team  members  should  work  issues  in  real  time  and  should  be  knowl¬ 
edgeable  of  their  OIPT  leader’s  assessment.  OIPT  and  other  staff  members 
should  minimize  requirements  for  the  program  manager  to  provide  pre-briefs 
independent  of  the  OIPT  process. 

•  10.5.  Role  of  Independent  Assessments 

Assessments,  independent  of  the  developer  and  the  user,  ensure  an  impartial 
evaluation  of  program  status.  However,  requirements  for  independent 
assessments  (for  example,  the  independent  cost  estimate  or  technology  readi¬ 
ness  assessment)  must  be  consistent  with  statutory  requirements  and  good 
management  practice.  Senior  acquisition  officials  should  consider  these 
assessments  when  making  acquisition  decisions.  Staff  offices  that  provide 
independent  assessments  should  support  the  orderly  and  timely  progression 
of  programs  through  the  acquisition  process.  IPTs  should  have  access  to 
independent  assessments  to  enable  full  and  open  discussion  of  issues. 

10.5.2.  Technology  Maturity  and  Technology  Readiness  Assessments 

Technology  maturity  is  a  measure  of  the  degree  to  which  proposed  critical 
technologies  meet  program  objectives;  and,  is  a  principal  element  of  program 
risk.  A  technology  readiness  assessment  examines  program  concepts,  tech¬ 
nology  requirements,  and  demonstrated  technology  capabilities  in  order  to 
determine  technological  maturity. 

The  program  manager  should  identify  critical  technologies  via  the  Work 
Breakdown  Structure.  In  order  to  provide  useful  technology  maturity  infor¬ 
mation  to  the  acquisition  review  process,  technology  readiness  assessments 
of  critical  technologies  and  identification  of  Critical  Program  Information 
(CPI)  must  be  completed  prior  to  Milestone  Decision  points  B  and  C. 

The  DoD  Component  Science  and  Technology  (S&T)  Executive  directs  the 
technology  readiness  assessment  and,  for  Acquisition  Category  ID  and 
Acquisition  Category  IAM  programs,  submits  the  findings  to  the  CAE  who 
should  submit  his  or  her  report  to  the  DUSD(S&T)  with  a  recommended 
technology  readiness  level  (TRL)  (or  some  equivalent  assessment)  for  each 
critical  technology.  When  the  DoD  Component  S&T  Executive  submits  his 
or  her  findings  to  the  CAE,  he  or  she  should  provide  the  DUSD(S&T)  an 
information  copy  of  those  findings.  In  cooperation  with  the  DoD  Component 
S&T  Executive  and  the  program  office,  the  DUSD(S&T)  should  evaluate  the 
technology  readiness  assessment  and,  if  he/she  concurs,  forward  findings  to 
the  OIPT  leader  and  DAB.  If  the  DUSD(S&T)  does  not  concur  with  the 
technology  readiness  assessment  findings,  an  independent  technology  readi¬ 
ness  assessment,  under  the  direction  of  the  DUSD(S&T),  should  be  required. 
A  summary  table  of  TRL  descriptions,  Table  10.5.2.1,  follows: 
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Table  10.5.2.1.  TRL  Descriptions 


Technology  Readiness  Level 

Description 

1 .  Basic  principles  observed  and 
reported. 

Lowest  level  of  technology  readiness.  Scientific  research  begins 
to  be  translated  into  applied  research  and  development.  Exam¬ 
ples  might  include  paper  studies  of  a  technology’s  basic 
properties. 

2.  Technology  concept  and/or  appli¬ 
cation  formulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical 
applications  can  be  invented.  Applications  are  speculative  and 
there  may  be  no  proof  or  detailed  analysis  to  support  the 
assumptions.  Examples  are  limited  to  analytic  studies. 

3.  Analytical  and  experimental  critical 
function  and/or  characteristic  proof  of 
concept. 

Active  research  and  development  is  initiated.  This  includes  ana¬ 
lytical  studies  and  laboratory  studies  to  physically  validate  ana¬ 
lytical  predictions  of  separate  elements  of  the  technology. 
Examples  include  components  that  are  not  yet  integrated  or 
representative. 

4.  Component  and/or  breadboard 
validation  in  laboratory  environment. 

Basic  technological  components  are  integrated  to  establish  that 
they  will  work  together.  This  is  relatively  “low  fidelity”  compared 
to  the  eventual  system.  Examples  include  integration  of  “ad  hoc” 
hardware  in  the  laboratory. 

5.  Component  and/or  breadboard 
validation  in  relevant  environment. 

Fidelity  of  breadboard  technology  increases  significantly.  The 
basic  technological  components  are  integrated  with  reasonably 
realistic  supporting  elements  so  it  can  be  tested  in  a  simulated 
environment.  Examples  include  “high  fidelity”  laboratory  integra¬ 
tion  of  components. 

6.  System/subsystem  model  or  pro¬ 
totype  demonstration  in  a  relevant 
environment. 

Representative  model  or  prototype  system,  which  is  well  beyond 
that  of  TRL  5,  is  tested  in  a  relevant  environment.  Represents  a 
major  step  up  in  a  technology’s  demonstrated  readiness.  Exam¬ 
ples  include  testing  a  prototype  in  a  high-fidelity  laboratory  envi¬ 
ronment  or  in  simulated  operational  environment. 

7.  System  prototype  demonstration  in 
an  operational  environment. 

Prototype  near,  or  at,  planned  operational  system.  Represents  a 
major  step  up  from  TRL  6,  requiring  demonstration  of  an  actual 
system  prototype  in  an  operational  environment  such  as  an 
aircraft,  vehicle,  or  space.  Examples  include  testing  the  proto¬ 
type  in  a  test  bed  aircraft. 

8.  Actual  system  completed  and 
qualified  through  test  and 
demonstration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under 
expected  conditions.  In  almost  all  cases,  this  TRL  represents  the 
end  of  true  system  development.  Examples  include  develop¬ 
mental  test  and  evaluation  of  the  system  in  its  intended  weapon 
system  to  determine  if  it  meets  design  specifications. 

9.  Actual  system  proven  through 
successful  mission  operations. 

Actual  application  of  the  technology  in  its  final  form  and  under 
mission  conditions,  such  as  those  encountered  in  operational 
test  and  evaluation.  Examples  include  using  the  system  under 
operational  mission  conditions. 
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The  use  of  TRLs  enables  consistent,  uniform,  discussions  of  technical  matur¬ 
ity  across  different  types  of  technologies.  Decision  authorities  will  consider 
the  recommended  TRLs  (or  some  equivalent  assessment  methodology,  e.g., 
Willoughby  templates)  when  assessing  program  risk.  TRLs  are  a  measure  of 
technical  maturity.  They  do  not  discuss  the  probability  of  occurrence  (i.e.,  the 
likelihood  of  attaining  required  maturity)  or  the  impact  of  not  achieving  tech¬ 
nology  maturity. 

A.4  REFERENCES 

•  From  DoDI  5000.2,  Operation  of  the  Defense  Acquisition  System,  dated 
May  12,  2003  (see  Subsections  A.2.2  and  A.2.3  of  this  appendix) 

-  (g)  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  3170.01  Series, 

“Requirements  Generation  System,”  April  15,  2001.  Note: 
DoDI  5000.2  actually  referenced  an  April  15,  2001,  version  of  this 
document.  The  most  current  version  of  this  manual,  CJCSI 
3 170.01  E,  Joint  Capabilities  Integration  and  Development  System, 
11  May  2005,  should  be  available  at  http://www.dtic.mil/ 
cjcs_directiveslcjcslmanuals.htm. 

-  (q)  Section  2364  of  title  10,  United  States  Code,  “Coordination  and 

Communication  of  Defense  Research  Activities.” 

-  (an)  Section  803,  Public  Law  107-314,  “Bob  Stump  National  Defense 

Authorization  Act  for  Fiscal  Year  2003,”  “Spiral  development 
under  major  defense  acquisition  programs.” 

-  (ar)  DoD  Instruction  4630.8,  “Procedures  for  Interoperability  and 

Supportability  of  Information  Technology  (IT)  and  National 
Security  Systems  (NSS),”  May  2,  2002 

-  (as)  DoD  Directive,  Number  4630.5,  “Interoperability  and 

Supportability  of  Information  Technology  (IT)  and  National 
Security  Systems  (NSS),”  January  11,  2002. 
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ACRONYMS  AND  ABBREVIATIONS  FOR  APPENDIX  A 


ACAT 

Acquisition  Category 

ADM 

Acquisition  Decision  Memorandum 

AoA 

Analysis  of  Alternatives 

AKSS 

AT&L  Knowledge  Sharing  System 

AT&L 

Acquisition,  Technology,  and  Logistics 

C4ISP 

Command,  Control,  Communications,  Computers,  and 
Intelligence  Support  Plan 

CAE 

Component  Acquisition  Executive 

CDD 

Capability  Development  Document 

CJCSI 

Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 

COTS 

commercial-off-the-shelf 

CPI 

Critical  Program  Information 

DAB 

Defense  Acquisition  Board 

DARC 

Defense  Acquisition  Resource  Center 

DAU 

Defense  Acquisition  University 

DDR&E 

Director  of  Defense  Research  and  Evaluation 

DoD 

Department  of  Defense 

DoDD 

Department  of  Defense  Directive 

DoDI 

Department  of  Defense  Instruction 

DOTMLPF 

doctrine,  organization,  training,  materiel,  leadership, 
personnel,  and  facilities 

DUSD(S&T) 

Deputy  Under  Secretary  of  Defense  for  Science  and 
Technology 

ICD 

Initial  Capabilities  Document 

ICE 

independent  cost  estimate 

IPPD 

Integrated  Product  and  Process  Development 

IPT 

Integrated  Product  Team 

IT 

Information  Technology 

ITA 

Independent  Technical  Assessment 
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MAIS 

Major  Automated  Information  System 

MDA 

Milestone  Decision  Authority 

MDAP 

Major  Defense  Acquisition  Program 

MS 

Milestone 

NEPA 

National  Environmental  Policy  Act 

NSS 

National  Security  Systems 

OIPT 

Overarching  Integrated  Product  Team 

PESHE 

Programmatic  Environment,  Safety,  and  Occupational 
Health  Evaluation 

PM 

Program  Manager 

S&T 

Science  and  Technology 

SDD 

System  Development  and  Demonstration 

T&E 

test  and  evaluation 

TDS 

Technology  Development  Strategy 

TRA 

Technology  Readiness  Assessment 

TRL 

Technology  Readiness  Level 
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APPENDIX  B. 

SUMMARY  OF  GOVERNMENT  ACCOUNTABILITY  OFFICE 
(GAO)42  REPORTS  AND  DEPARTMENT  OF  DEFENSE  (DoD) 

IMPLEMENTATION 

B.l  GAO  Reports  . B-3 

B.2  GAO  Recommendations  . B-6 

B.3  DoD  Comments  and  GAO  Evaluation . B-8 

B.4  References . B-10 

Acronyms  and  Abbreviations  for  Appendix  B  . B-ll 


Formerly  the  General  Accounting  Office  (GAO).  Effective  July  7,  2004,  the  GAO’s  legal  name 
became  the  Government  Accountability  Office. 
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Several  GAO  reports  addressed  the  DoD  acquisition  system  and  made  recommen¬ 
dations  that  influenced  the  DoD  5000  series  of  publications.  The  following  presents  a 
brief  summary  of  GAO-related  work,  along  with  references  for  the  source  documents. 

B.l  GAO  REPORTS 

The  subcommittee  on  Readiness  and  Management  Support  of  the  Committee  on 
Armed  Services,  U.S.  Senate,  which  has  oversight  on  acquisitions  policy,  enlisted  the 
GAO  in  a  study  of  best  commercial  practices  related  to  defense  acquisition.  A  series  of 
GAO  reports  and  related  testimony  assessed  how  best  commercial  practices  could 
improve  the  way  DoD  incorporates  new  technology  into  weapon  system  programs  and 
reduces  risk.  These  reports,  issued  from  1996-2000  (the  principals  of  which  are  listed  as 
References  1,  2,  3),  offered  DoD  some  guidance  and  had  significant  influence  on  the 
current  versions  of  the  DoD  5000  series  of  documents:  Department  of  Defense  Directive 
(DoDD)  5000.1,  Department  of  Defense  Instruction  (DoDI)  5000.2,  and  the  Defense 
Acquisition  Guidebook  (Refs.  4,  5,  6). 

The  weapon  system  acquisition  cycle  for  DoD  major  weapon  systems  before  the 
incorporation  of  best  commercial  practices  could  be  illustrated  as  shown  in  Figure  B-l.43 
Technology,  design,  and  manufacturing  knowledge  was  obtained  concurrently. 


Program  Begin  product 

launch  development 


Technology 


Knowledge 

attainment 


Manufacturing 


Design 


Figure  B-1.  DoD’s  Current  Weapon  System  Acquisition 
(Source:  Ref.  3) 


43  Figure  B-l  depicts  the  weapon  system  acquisition  cycle  for  DoD  major  weapon  systems  in  the  year 
2000.  Figure  B-5  outlines  the  current  Defense  Acquisition  Management  Framework. 
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The  major  GAO  recommendation,  which  followed  best  commercial  practice,  was 
to  minimize  technology  development  during  product  development  and  match  require¬ 
ments  with  technological  capability  before  product  development  is  launched.  Proof  that 
the  technology  will  work  and  can  be  demonstrated  to  a  high  level  of  maturity  is  critical  to 
lowering  risk  and  avoiding  large  cost  overruns.  Associated  with  this  principle  are  the 
needs  to  develop  high  standards  for  finding  the  maturity  and  readiness  of  technology,  to 
establish  disciplined  paths  that  technology  must  take  to  be  included  in  products,  and  to 
provide  strong  gatekeepers  to  decide  when  to  allow  the  technology  into  a  product  devel¬ 
opment  program.  GAO  recommended  that  DoD  not  launch  a  program  until  the  technolo¬ 
gies  needed  to  meet  a  new  weapons  requirement  are  mature.  To  separate  this  technology 
development  from  the  program,  GAO  best  practices’  recommendations  suggest  that  a 
technology  and  concept  maturation  phase  follow  concept  exploration  and  precede  pro¬ 
gram  launch,  as  illustrated  in  Figure  B-2. 


Concept 

Technology  and  Concept 

Exploration 

Maturation 

▲  ▲  ▲ 


Need  Concept  Technology 

recognition  selected  matches  need 

Figure  B-2.  Weapon  Acquisition  Phases  That  Should  Precede 
the  Launch  of  a  New  Program 
(Source:  Ref.  3) 

The  GAO  review  of  best  practices  for  including  new  technology  in  products  (see 
Ref.  2)  applied  a  scale  of  Technology  Readiness  Levels  (TRLs)  pioneered  by  the 
National  Aeronautics  and  Space  Administration  (NASA)  and  adapted  by  the  Air  Force 
Research  Laboratory  (AFRL).  “TRLs  proved  to  be  reliable  indicators  of  the  relative 
maturity  of  the  23  technologies  reviewed,  both  commercial  and  military,  and  their  even¬ 
tual  success  after  they  were  included  in  product  development  programs”  (Ref.  2,  p.  22). 

To  show  that  a  design  is  mature,  the  GAO  studies  suggest  that  a  product  develop¬ 
ment  phase  should  include  a  distinct  system  integration  effort  before  system  demonstra¬ 
tion  effort  to  demonstrate  the  effectiveness  of  the  product  and  processes  (see  Figure  B-3). 

Figure  B-4  shows  GAO’s  final  proposal  for  a  potential  DoD  technology  and  pro¬ 
duct  development  process  based  on  commercial  best  practices.  It  should  be  noted  that 
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Figure  B-3.  Product  Development  Phase  To  Deliver  a 
Mature  Design  and  Key  Processes 
(Source:  Ref.  3) 
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Figure  B-4.  Potential  DoD  Technology  and  Product  Development  Process 
Incorporating  Best  Practices 
(Source:  Ref.  3) 

leading  commercial  firms  launch  a  new  product  later — after  technology  is  complete — 
than  DoD  launches  a  new  product.  Paragraphs  B.2  and  B.3  of  this  appendix  provide  the 
GAO  recommendations  for  DoD  management  of  Technology  Development  and  the  DoD 
response  as  reported  in  Reference  2.  DoD  did  not  agree  entirely  with  GAO’s  recommen¬ 
dations  and  is  willing  to  accept  more  risk.  DoD  considered  TRL  6  as  an  acceptable 
readiness-level  risk  for  a  weapon  system  entering  the  program  definition  stage  (see 
Figure  B-l)  and  TRL  7  as  an  acceptable  readiness-level  risk  for  the  Engineering  and 
Manufacturing  Development  (EMD)  stage.  GAO  accepted  this. 

Figure  B-5  outlines  the  current  Defense  Acquisition  Management  Framework. 
The  relationship  to  the  GAO  recommendation  of  Figure  B-4  is  evident. 
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Figure  B-5.  Defense  Acquisition  Management  Framework 
(Source:  Ref.  5) 


B.2  GAO  RECOMMENDATIONS 

The  following  paragraphs  are  direct  quotations  from  Chapter  5  of  Reference  2: 

We  have  previously  recommended  that  DOD  separate  technology  devel¬ 
opment  from  weapon  system  programs.  That  recommendation  was  made 
without  prejudice  toward  the  necessity  of  technology  development  but 
rather  with  the  intent  that  programs  could  be  better  managed  if  such  devel¬ 
opment  was  conducted  outside  of  a  program  manager’s  purview.  Simi¬ 
larly,  the  recommendations  that  follow  are  made  without  prejudice  toward 
or  the  intention  of  compromising-the  basic  research  and  other  activities 
that  S&T  organizations  perform.  We  recognize  that  implementation  of 
these  recommendations  will  have  organizational,  funding,  and  process 
implications  and  will  require  the  cooperation  of  the  Congress,  (p.  63) 

To  help  ensure  that  new  technologies  are  vigorously  pursued  and  success¬ 
fully  moved  into  weapon  system  programs,  we  recommend  that  the  Sec¬ 
retary  of  Defense  adopt  a  disciplined  and  knowledge-based  method  for 
assessing  technology  maturity,  such  as  TRLs,  DOD-wide.  This  practice 
should  employ  standards  for  assessing  risks  of  handoff  to  program  man¬ 
agers  that  are  based  on  a  technology’s  level  of  demonstration  and  its  criti¬ 
cality  to  meeting  the  weapon  system’s  requirements,  (p.64) 

With  these  tools  in  hand,  we  recommend  that  the  Secretary  (1)  establish 
the  place  at  which  a  match  is  achieved  between  key  technologies  and 
weapon  system  requirements  as  the  proper  time  for  committing  to  the  cost, 
schedule,  and  performance  baseline  for  developing  and  producing  that 
weapon  system  and  (2)  require  that  key  technologies  reach  a  high  maturity 
level — analogous  to  TRL  7 — before  making  that  commitment.  This  would 
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approximate  the  launch  point  for  product  development  as  practiced  by 
leading  commercial  firms,  (p.  64) 

We  recommend  that  the  Secretary  find  ways  to  ensure  that  the  managers 
responsible  for  maturing  the  technologies  and  designing  weapon  systems 
before  product  development  are  provided  the  more  flexible  environment 
that  is  suitable  for  the  discovery  of  knowledge,  as  distinct  from  the  deliv¬ 
ery  of  a  product.  Providing  more  flexibility  will  require  the  cooperation  of 
requirements  managers  and  resource  managers  so  that  rigid  requirements 
or  the  threat  of  jeopardizing  the  funding  planned  to  start  product  develop¬ 
ment  will  not  put  pressure  on  program  managers  to  accept  immature  tech¬ 
nologies.  Such  an  environment  may  not  be  feasible  if  the  program 
definition  and  risk  reduction  phase  remains  the  effective  launch  point  for 
an  entire  weapon  system  program,  (p.  64) 

An  implication  of  these  recommendations  is  that  S&T  organizations  will 
have  to  play  a  greater  role  in  maturing  technologies  to  higher  levels  and 
should  be  funded  accordingly.  Therefore,  we  recommend  that  the  Secre¬ 
tary  of  Defense  evaluate  the  different  ways  S&T  organizations  can  play  a 
greater  role  in  helping  technologies  reach  high  levels  of  maturity  before 
product  development  begins.  For  example,  given  that  a  technology  has 
sufficient  potential  for  application  to  a  weapon  system,  at  a  minimum,  an 
S&T  organization  should  be  responsible  for  taking  a  technology  to  TRL  6 
before  it  is  handed  off  to  a  program  office  at  the  program  definition  and 
risk  reduction  phase.  During  this  phase,  the  program  manager  would  be 
responsible  for  maturing  the  technology  to  TRL  7  before  it  is  included  in 
an  engineering  and  manufacturing  development  program.  In  a  situation 
where  a  single,  design-pacing  technology  is  to  be  developed  for  a  known 
application — like  the  nonpenetrating  periscope — an  S&T  organization 
should  be  required  to  mature  that  technology  to  TRL  7  before  it  is  turned 
over  to  a  product  development  manager.  S&T  organizations  could  play  a 
similar  role  when  a  significant  new  technology  is  being  prepared  for 
insertion  into  an  existing  weapon  system.  Finally,  when  multiple  new 
technologies  are  to  be  merged  to  create  a  weapon  system,  S&T  organiza¬ 
tions  should  be  required  to  bring  key  technologies  to  TRL  6  and  then 
become  part  of  a  hybrid  organization  with  product  developers  to  integrate 
the  technologies  and  bring  them  to  TRL  7  before  handing  full  responsibil¬ 
ity  to  a  product  development  manager,  (pp.  64-65) 

To  help  guard  against  the  possibility  that  the  more  basic  research  and  tech¬ 
nology  development  activities  would  be  compromised  by  having  S&T 
organizations  routinely  take  key  technologies  to  TRL  6  or  higher,  we  rec¬ 
ommend  that  the  Secretary  extract  lessons  from  the  nonpenetrating  peri¬ 
scope,  the  AAAV,  and  the  Army’s  Future  Scout  programs,  and  other  ATD 
and  ACTD  programs.  Specifically,  the  Secretary  should  assess  whether 
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the  resources  needed  to  enable  S&T  organizations  to  play  a  leading  role  in 
the  development  of  technologies  and,  in  some  cases,  preliminary  system 
design,  detracted  from  or  displaced  more  basic  research  and  technology 
development  programs,  (p.  65) 

Finally,  we  recommend  that  the  Secretary  empower  managers  of  product 
development  programs  to  refuse  to  accept  key  technologies  with  low  lev¬ 
els  of  demonstrated  maturity.  The  Secretary  can  encourage  this  behavior 
through  supportive  decisions  on  individual  programs,  such  as  by  denying 
proposals  to  defer  the  development  of  key  technologies  and  by  favoring 
proposals  to  lengthen  schedules  or  lessen  requirements  to  reduce  techno¬ 
logical  risk  early,  (p.  65) 

B.3  DoD  COMMENTS  AND  GAO  EVALUATION 

The  following  paragraphs  are  direct  quotations  from  Chapter  5  of  Reference  2: 

DOD  generally  concurred  with  a  draft  of  this  report  and  its  recommenda¬ 
tions,  noting  that  the  traditional  path  to  new  weapon  system  development 
is  no  longer  affordable  or  necessary  (App.  I).44  DOD  stated  that  it  has 
embarked  upon  a  “Revolution  in  Business  Affairs”  that  will  enable  new 
technologies  to  be  developed  more  efficiently  and  effectively.  It  believes 
that  the  first  steps  in  this  direction  have  already  been  taken  but  agrees  that 
more  progress  needs  to  be  made.  DOD  agreed  that  TRLs  are  necessary  in 
assisting  decision-makers  in  deciding  on  when  and  where  to  insert  new 
technologies  into  weapon  system  programs  and  that  weapon  system  man¬ 
agers  should  ensure  that  technology  is  matured  to  a  TRL  7  before  insertion 
occurs.  DOD  concurred  that  S&T  organizations  should  be  involved  in 
maturing  technologies  to  high  levels,  such  as  TRL  6,  before  transitioning 
to  the  engineering  and  manufacturing  development  phase  and  agreed  to 
assess  the  impact  of  this  involvement  on  other  S&T  resources.  We  note 
that  the  best  practice  is  to  mature  technology  to  at  least  a  TRL  7  before 
starting  the  engineering  and  manufacturing  development  phase,  whether 
the  technology  is  managed  by  an  S&T  organization,  a  weapon  system  pro¬ 
gram  manager,  or  a  hybrid  of  the  two  organizations,  (pp.  65-66) 

DOD  noted  that  while  TRLs  are  important  and  necessary,  the  increasing 
projected  life  for  new  weapon  systems,  total  ownership  costs,  and  urgency 
based  upon  threat  assessments  are  also  important  considerations  for 
system  development  decisions.  We  agree  and  note  that  our  recommen¬ 
dations  are  not  intended  to  cover  all  aspects  of  weapon  system  develop¬ 
ment  decisions  or  to  suggest  that  technology  maturity  is  the  only  factor  in 


44  Appendix  I  of  GAO/NSIAD-99-162  is  called  “Technology  Readiness  Levels  and  Their  Definitions.” 
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such  decisions.  Rather,  the  recommendations  are  in  keeping  with  the 
purpose  of  the  report,  “to  determine  whether  best  practices  offer  methods 
to  improve  the  way  DOD  matures  new  technology  so  that  it  can  be  assimi¬ 
lated  into  weapon  system  programs  with  less  disruption.”  We  believe  that 
a  knowledge-based  approach  to  maturing  technology,  such  as  TRLs,  can 
benefit  other  considerations  as  well.  For  example,  decisions  on  what 
technologies  to  include  in  a  weapon  system  and  when  to  include  them  can 
have  a  significant  bearing  on  its  total  ownership  costs,  (pp.  66) 

DOD  stated  that  there  should  be  an  established  point  for  the  transition  of 
technologies  and  that  it  plans  to  supplement  its  milestone  review  process 
with  additional  guidance  in  the  next  revisions  to  DOD  5000. 2-R.45  It  also 
stated  that  its  policy  on  the  evolutionary  approach  to  weapon  acquisitions 
should  be  developed  in  consonance  with  the  technology  transition  strat¬ 
egy.  We  cannot  comment  on  the  revisions  to  the  directive  or  the  evolu¬ 
tionary  acquisition  policy  because  they  have  yet  to  be  published. 
However,  under  the  current  milestone  review  process,  the  pressures  placed 
on  a  program  during  the  program  definition  and  risk  reduction  phase — 
when  much  technology  development  occurs — can  operate  against  the 
flexibility  and  judgments  that  are  needed  to  mature  technologies.  If  the 
revisions  to  the  directive  supplement  the  current  milestones  without 
relieving  the  pressures  brought  to  bear  on  programs  as  they  are  launched 
in  the  program  definition  and  risk  reduction  phase,  it  will  remain  difficult 
to  discourage  the  acceptance  of  immature  technologies  in  the  design  of 
new  weapon  systems.  To  relieve  these  pressures,  we  encourage  DOD,  as  it 
develops  the  directive  and  the  evolutionary  acquisition  policy,  to  separate 
technology  development  from  product  development  and  to  redefine  the 
launch  point  for  a  program  as  the  point  at  which  enough  knowledge  has 
been  gained  to  ensure  that  a  match  is  reached  between  the  maturity  of  key 
technologies  and  weapon  system  requirements,  (pp.  66-67) 

DOD  also  stated  that  program  managers  already  have  the  ability  to  reject 
inappropriately  mature  technologies,  and  to  the  extent  technology  imma¬ 
turity  affects  acquisition  baselines,  to  advise  acquisition  executives  of  fea¬ 
sible  alternatives.  We  did  not  find  this  to  be  the  case  in  our  review.  Rather, 
we  found  that  the  program  managers’  ability  to  reject  immature  technolo¬ 
gies  is  hampered  by  (1)  untradable  requirements  that  force  acceptance  of 
technologies  despite  their  immaturity  and  (2)  reliance  on  tools  forjudging 
technology  maturity  that  fail  to  alert  the  managers  of  the  high  risks  that 
would  prompt  such  a  rejection.  As  noted  in  the  report,  once  a  weapon 
system  program  begins,  the  environment  becomes  inflexible  and 


45  DoD  5000. 2R  was  revised  and  evolved  into  the  Interim  Defense  Acquisition  Guidebook  (October 
2002).  The  Interim  Defense  Acquisition  Guidebook  was  then  revised  and  is  now  called  the  Defense 
Acquisition  Guidebook,  October  2004. 
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deviations  to  program  baselines  can  attract  unwanted  attention.  This 
reality  limits  the  program  managers’  ability  to  reject  immature 
technologies,  (p.  67) 
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ACRONYMS  AND  ABBREVIATIONS  FOR  APPENDIX  B 


AAAV 

Advanced  Amphibious  Assault  Vehicle 

ACTD 

Advanced  Concept  Technology  Demonstration 

AFRL 

Air  Force  Research  Faboratory 

AKSS 

AT&F  Knowledge  Sharing  System 

ATD 

Advanced  Technology  Demonstration 

DARC 

Defense  Acquisition  Resource  Center 

DAU 

Defense  Acquisition  University 

DoD,  DOD 

Department  of  Defense 

DoDD 

Department  of  Defense  Directive 

DoDI 

Department  of  Defense  Instruction 

EMD 

Engineering  and  Manufacturing  Development 

FOC 

full  operational  capability 

GAO 

General  Accounting  Office 

Government  Accountability  Office 

IOC 

initial  operational  capability 

LRIP 

Fow  Rate  Initial  Production 

NASA 

National  Aeronautics  and  Space  Administration 

NSIAD 

National  Security  and  International  Affairs  Division  (GAO) 

S&T 

Science  and  Technology 

TRL 

Technology  Readiness  Fevel 
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C.l  OVERVIEW:  TECHNOLOGY  READINESS  LEVEL  (TRL)  CONCEPT 


The  Defense  Acquisition  Guidebook  establishes  technology  maturity  as  “a  meas¬ 
ure  of  the  degree  to  which  proposed  critical  technologies  meet  program  objectives.  A 
Technology  Readiness  Assessment  [TRA]  examines  program  concepts,  technology 
requirements,  and  demonstrated  technology  capabilities  in  order  to  determine  technologi¬ 
cal  maturity”  (Section  10.5.2.).  The  TRA  results  in  a  recommended  readiness  level  (i.e., 
TRL)  for  the  Critical  Technology  Elements  (CTEs)  being  evaluated. 

Using  TRLs  to  describe  maturity  of  technology  elements  originated  with  the 
National  Aeronautics  and  Space  Administration  (NASA)  in  the  1980s.  The  levels 
spanned  the  earliest  stages  of  scientific  investigation  (Level  1)  to  successful  use  in  a  sys¬ 
tem  (Level  9),  which  typically  means  having  successfully  flown  in  space  for  NASA.  The 
Department  of  Defense  (DoD)  has  adopted  the  NASA  definitions— with  only  minor 
modifications— for  the  nine  TRLs. 

TRLs  are  not  a  measure  of  design  validity.  CTEs  should  be  identified  and 
assessed  under  the  assumption  that  the  design,  developed  as  part  of  the  systems  engi¬ 
neering  approach,  is  adequate  for  the  performance  of  the  required  functions.  However, 
supporting  TRL  5  or  higher  without  a  detailed  design  or  architecture  is  difficult  and 
problematic. 

A  CTE  is  classified  as  either  a  hardware,  software,  or  a  manufacturing  technol¬ 
ogy.  The  remainder  of  this  appendix  discusses  best  practices  and  provides  examples  for 
assessing  technology  maturity  for  each  of  the  three  classes  of  technology.46 

C.1.1  The  TRL  Concept  for  Hardware 

Many  TRAs  evaluate  hardware  CTEs  that  are  being  developed  for  weapons  sys¬ 
tems,  communications  systems,  soldier  systems,  and  so  forth.  In  evaluating  hardware,  a 
strong  grasp  of  the  TRL  concept  is  important.  Table  C-l  shows  the  TRLs  used  to  assess 
hardware.  It  also  lists  typical  documentation  that  should  be  extracted  or  referenced  to 
support  a  TRL  assignment.  Table  C-2  includes  a  set  of  additional  definitions  that  help 
provide  a  uniform  interpretation  of  the  levels. 


46  Development  and  use  of  TRLs  for  medical-related  items,  specifically  drugs,  vaccines,  and  medical 
devices  must  adhere  to  Food  and  Drug  Administration  (FDA)  and  DoD  statutes  and  policy.  In 
recognition  of  this  situation,  the  Army  took  the  initiative  to  establish  biomedical  TRLs,  which  have 
been  included  in  Appendix  H. 
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Table  C-1.  Hardware  TRL  Definitions,  Descriptions,  and  Supporting  Information 
(Source:  Defense  Acquisition  Guidebook) 


TRL 

Definition 

Description 

Supporting  Information 

1 

Basic  principles 
observed  and 
reported. 

Lowest  level  of  technology  readi¬ 
ness.  Scientific  research  begins 
to  be  translated  into  applied 
research  and  development  (R&D). 
Examples  might  include  paper 
studies  of  a  technology’s  basic 
properties. 

Published  research  that  identifies  the  prin¬ 
ciples  that  underlie  this  technology.  Refer¬ 
ences  to  who,  where,  when. 

2 

Technology  con¬ 
cept  and/or  appli¬ 
cation  formulated. 

Invention  begins.  Once  basic 
principles  are  observed,  practical 
applications  can  be  invented. 
Applications  are  speculative,  and 
there  may  be  no  proof  or  detailed 
analysis  to  support  the  assump¬ 
tions.  Examples  are  limited  to 
analytic  studies. 

Publications  or  other  references  that  out¬ 
line  the  application  being  considered  and 
that  provide  analysis  to  support  the 
concept. 

3 

Analytical  and 
experimental 
critical  function 
and/or  character¬ 
istic  proof  of 
concept. 

Active  R&D  is  initiated.  This 
includes  analytical  studies  and 
laboratory  studies  to  validate 
physically  the  analytical  predic¬ 
tions  of  separate  elements  of  the 
technology.  Examples  include 
components  that  are  not  yet  inte¬ 
grated  or  representative. 

Results  of  laboratory  tests  performed  to 
measure  parameters  of  interest  and  com¬ 
parison  to  analytical  predictions  for  critical 
subsystems.  References  to  who,  where, 
and  when  these  tests  and  comparisons 
were  performed. 

4 

Component 
and/or  bread¬ 
board  validation 
in  a  laboratory 
environment. 

Basic  technological  components 
are  integrated  to  establish  that 
they  will  work  together.  This  is 
relatively  “low  fidelity”  compared 
with  the  eventual  system.  Exam¬ 
ples  include  integration  of  “ad 
hoc”  hardware  in  the  laboratory. 

System  concepts  that  have  been  consid¬ 
ered  and  results  from  testing  laboratory- 
scale  breadboard(s).  References  to  who 
did  this  work  and  when.  Provide  an  esti¬ 
mate  of  how  breadboard  hardware  and 
test  results  differ  from  the  expected  system 
goals. 

5 

Component  and / 
or  breadboard 
validation  in  a 
relevant 
environment. 

Fidelity  of  breadboard  technology 
increases  significantly.  The  basic 
technological  components  are 
integrated  with  reasonably  realis¬ 
tic  supporting  elements  so  they 
can  be  tested  in  a  simulated  envi¬ 
ronment.  Examples  include  “high- 
fidelity”  laboratory  integration  of 
components. 

Results  from  testing  a  laboratory  bread¬ 
board  system  are  integrated  with  other 
supporting  elements  in  a  simulated  opera¬ 
tional  environment.  How  does  the  “relevant 
environment”  differ  from  the  expected 
operational  environment?  How  do  the  test 
results  compare  with  expectations?  What 
problems,  if  any,  were  encountered?  Was 
the  breadboard  system  refined  to  more 
nearly  match  the  expected  system  goals? 
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Table  C-1.  Hardware  TRL  Definitions,  Descriptions,  and  Supporting  Information 
(Source:  Defense  Acquisition  Guidebook)  (Continued) 


TRL 

Definition 

Description 

Supporting  Information 

6 

System/subsystem  model  or 
prototype  demonstration  in  a 
relevant  environment. 

Representative  model  or  proto¬ 
type  system,  which  is  well 
beyond  that  of  TRL  5,  is  tested 
in  a  relevant  environment.  Rep¬ 
resents  a  major  step  up  in  a 
technology’s  demonstrated 
readiness.  Examples  include 
testing  a  prototype  in  a  high- 
fidelity  laboratory  environment 
or  in  a  simulated  operational 
environment. 

Results  from  laboratory  testing 
of  a  prototype  system  that  is 
near  the  desired  configuration 
in  terms  of  performance, 
weight,  and  volume.  How  did 
the  test  environment  differ  from 
the  operational  environment? 
Who  performed  the  tests?  How 
did  the  test  compare  with 
expectations?  What  problems, 
if  any,  were  encountered? 

What  are/were  the  plans, 
options,  or  actions  to  resolve 
problems  before  moving  to  the 
next  level? 

7 

System  prototype  demon¬ 
stration  in  an  operational 
environment. 

Prototype  near  or  at  planned 
operational  system.  Represents 
a  major  step  up  from  TRL  6  by 
requiring  demonstration  of  an 
actual  system  prototype  in  an 
operational  environment  (e.g.,  in 
an  aircraft,  in  a  vehicle,  or  in 
space).  Examples  include 
testing  the  prototype  in  a  test 
bed  aircraft. 

Results  from  testing  a  proto¬ 
type  system  in  an  operational 
environment.  Who  performed 
the  tests?  How  did  the  test 
compare  with  expectations? 

What  problems,  if  any,  were 
encountered?  What  are/were 
the  plans,  options,  or  actions  to 
resolve  problems  before 
moving  to  the  next  level? 

8 

Actual  system  completed 
and  qualified  through  test 
and  demonstration. 

Technology  has  been  proven  to 
work  in  its  final  form  and  under 
expected  conditions.  In  almost 
all  cases,  this  TRL  represents 
the  end  of  true  system  develop¬ 
ment.  Examples  include  devel¬ 
opmental  test  and  evaluation  of 
the  system  in  its  intended  wea¬ 
pon  system  to  determine  if  it 
meets  design  specifications. 

Results  of  testing  the  system  in 
its  final  configuration  under  the 
expected  range  of  environ¬ 
mental  conditions  in  which  it 
will  be  expected  to  operate. 
Assessment  of  whether  it  will 
meet  its  operational  require¬ 
ments.  What  problems,  if  any, 
were  encountered?  What 
are/were  the  plans,  options,  or 
actions  to  resolve  problems 
before  finalizing  the  design? 

9 

Actual  system  proven 
through  successful  mission 
operations. 

Actual  application  of  the  tech¬ 
nology  in  its  final  form  and  under 
mission  conditions,  such  as 
those  encountered  in  opera¬ 
tional  test  and  evaluation 
(OT&E).  Examples  include  using 
the  system  under  operational 
mission  conditions. 

OT&E  reports. 
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Table  C-2.  Additional  Definitions  of  TRL  Descriptive  Terms 
(Source:  Defense  Acquisition  Guidebook) 


Term 

Definition 

Breadboard 

Integrated  components  that  provide  a  representation  of  a  system/ 
subsystem  and  that  can  be  used  to  determine  concept  feasibility 
and  to  develop  technical  data.  Typically  configured  for  laboratory 
use  to  demonstrate  the  technical  principles  of  immediate  interest. 

May  resemble  final  system/subsystem  in  function  only. 

High  Fidelity 

Addresses  form,  fit,  and  function.  A  high-fidelity  laboratory  environ¬ 
ment  would  involve  testing  with  equipment  that  can  simulate  and 
validate  all  system  specifications  within  a  laboratory  setting. 

Low  Fidelity 

A  representative  of  the  component  or  system  that  has  limited  ability 
to  provide  anything  but  first-order  information  about  the  end  product. 
Low-fidelity  assessments  are  used  to  provide  trend  analysis. 

Model 

A  functional  form  of  a  system,  generally  reduced  in  scale,  near  or  at 
operational  specification.  Models  will  be  sufficiently  hardened  to 
allow  demonstration  of  the  technical  and  operational  capabilities 
required  of  the  final  system. 

Operational  Environment 

Environment  that  addresses  all  the  operational  requirements  and 
specifications  required  of  the  final  system  to  include  platform/ 
packaging. 

Prototype 

A  physical  or  virtual  model  used  to  evaluate  the  technical  or  manu¬ 
facturing  feasibility  or  military  utility  of  a  particular  technology  or 
process,  concept,  end  item,  or  system. 

Relevant  Environment 

Testing  environment  that  simulates  the  key  aspects  of  the  opera¬ 
tional  environment. 

Simulated  Operational  Environment 

Either  (1)  a  real  environment  that  can  simulate  all  the  operational 
requirements  and  specifications  required  of  the  final  system  or  (2)  a 
simulated  environment  that  allows  for  testing  of  a  virtual  prototype. 
Used  in  either  case  to  determine  whether  a  developmental  system 
meets  the  operational  requirements  and  specifications  of  the  final 
system. 

C.1.2  The  TRL  Concept  for  Software 

Hardware  technology  may  include  software  that  executes  on  the  hardware  if 
(1)  the  software  is  not  being  developed  or  modified  as  part  of  the  acquisition  or  (2)  the 
software  is  not  the  reason  for  placing  the  element  on  the  CTE  list.  If  the  system  engi¬ 
neering  process  develops  the  software  and  the  software  is  a  CTE,  it  should  appear  as  a 
software  CTE— with  the  hardware  appearing  as  a  hardware  CTE. 

Table  C-3  shows  the  TRLs  used  to  assess  software.  These  TRLs  are  a  consolida¬ 
tion  of  the  software  TRLs  used  by  the  Navy  and  the  Army  and  approved  by  the  Infor¬ 
mation  Technology  TRL  Working  Group.  Although  the  overall  definitions  are  similar  to 
the  TRLs  for  hardware,  the  examples  and  the  documentation  needed  to  support  the 
assessment  differ. 
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Table  C-3.  Software  TRL  Definitions,  Descriptions,  and  Supporting  Information 
(Source:  IT  TRL  Working  Group  Minutes,  November  9,  2004) 


TRL 

Definition 

Description 

Supporting  Information 

1 

Basic  principles  observed 
and  reported. 

Lowest  level  of  software  technol¬ 
ogy  readiness.  A  new  software 
domain  is  being  investigated  by 
the  basic  research  community. 

This  level  extends  to  the  devel¬ 
opment  of  basic  use,  basic  prop¬ 
erties  of  software  architecture, 
mathematical  formulations,  and 
general  algorithms 

Basic  research  activities, 
research  articles,  peer-reviewed 
white  papers,  point  papers,  early 
lab  model  of  basic  concept  may 
be  useful  for  substantiating  the 
TRL  level. 

2 

Technology  concept 
and/or  application 
formulated. 

Once  basic  principles  are 
observed,  practical  applications 
can  be  invented.  Applications  are 
speculative,  and  there  may  be  no 
proof  or  detailed  analysis  to  sup¬ 
port  the  assumptions.  Examples 
are  limited  to  analytic  studies 
using  synthetic  data. 

Applied  research  activities,  ana¬ 
lytic  studies,  small  code  units, 
and  papers  comparing 
competing  technologies. 

3 

Analytical  and  experi¬ 
mental  critical  function 
and/or  characteristic  proof 
of  concept. 

Active  R&D  is  initiated.  The  level 
at  which  scientific  feasibility  is 
demonstrated  through  analytical 
and  laboratory  studies.  This  level 
extends  to  the  development  of 
limited  functionality  environments 
to  validate  critical  properties  and 
analytical  predictions  using  non- 
integrated  software  components 
and  partially  representative  data. 

Algorithms  run  on  a  surrogate 
processor  in  a  laboratory  envi¬ 
ronment,  instrumented  compo¬ 
nents  operating  in  laboratory 
environment,  laboratory  results 
showing  validation  of  critical 
properties. 

4 

Module  and/or  subsystem 
validation  in  a  laboratory 
environment  (i.e.,  software 
prototype  development 
environment). 

Basic  software  components  are 
integrated  to  establish  that  they 
will  work  together.  They  are  rela¬ 
tively  primitive  with  regard  to 
efficiency  and  robustness  com¬ 
pared  with  the  eventual  system. 
Architecture  development  initi¬ 
ated  to  include  interoperability, 
reliability,  maintainability,  exten¬ 
sibility,  scalability,  and  security 
issues.  Emulation  with  current/ 
legacy  elements  as  appropriate. 
Prototypes  developed  to  dem¬ 
onstrate  different  aspects  of 
eventual  system. 

Advanced  technology  develop¬ 
ment,  stand-alone  prototype 
solving  a  synthetic  full-scale 
problem,  or  standalone  proto¬ 
type  processing  fully  represen¬ 
tative  data  sets. 
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Table  C-3.  Software  TRL  Definitions,  Descriptions,  and  Supporting  Information 
(Source:  IT  TRL  Working  Group  Minutes,  November  9,  2004)  (Continued) 


TRL 

Definition 

Description 

Supporting  Information 

5 

Module  and/or  subsystem 
validation  in  a  relevant 
environment. 

Level  at  which  software  tech¬ 
nology  is  ready  to  start  integra¬ 
tion  with  existing  systems.  The 
prototype  implementations  con¬ 
form  to  target  environment/ 
interfaces.  Experiments  with 
realistic  problems.  Simulated 
interfaces  to  existing  systems. 
System  software  architecture 
established.  Algorithms  run  on  a 
processor(s)  with  characteristics 
expected  in  the  operational 
environment. 

System  architecture  diagram 
around  technology  element  with 
critical  performance  require¬ 
ments  defined.  Processor  selec¬ 
tion  analysis,  Simulation/ 
Stimulation  (Sim/Stim)  Labora¬ 
tory  buildup  plan.  Software 
placed  under  configuration  man¬ 
agement.  COTS/GOTS  in  the 
system  software  architecture  are 
identified. 

6 

Module  and/or  subsystem 
validation  in  a  relevant 
end-to-end  environment. 

Level  at  which  the  engineering 
feasibility  of  a  software  technol¬ 
ogy  is  demonstrated.  This  level 
extends  to  laboratory  prototype 
implementations  on  full-scale 
realistic  problems  in  which  the 
software  technology  is  partially 
integrated  with  existing  hard¬ 
ware/software  systems. 

Results  from  laboratory  testing 
of  a  prototype  package  that  is 
near  the  desired  configuration  in 
terms  of  performance,  including 
physical,  logical,  data,  and  secu¬ 
rity  interfaces.  Comparisons 
between  tested  environment  and 
operational  environment  analyti¬ 
cally  understood.  Analysis  and 
test  measurements  quantifying 
contribution  to  system-wide 
requirements  such  as  through¬ 
put,  scalability,  and  reliability. 
Analysis  of  human-computer 
(user  environment)  begun. 

7 

System  prototype  demon¬ 
stration  in  an  operational 
high-fidelity  environment. 

Level  at  which  the  program  fea¬ 
sibility  of  a  software  technology  is 
demonstrated.  This  level  extends 
to  operational  environment  proto¬ 
type  implementations  where  criti¬ 
cal  technical  risk  functionality  is 
available  for  demonstration  and  a 
test  in  which  the  software  tech¬ 
nology  is  well  integrated  with 
operational  hardware/software 
systems. 

Critical  technological  properties 
are  measured  against  require¬ 
ments  in  a  simulated  operational 
environment. 

8 

Actual  system  completed 
and  mission  qualified 
through  test  and  demon¬ 
stration  in  an  operational 
environment. 

Level  at  which  a  software  tech¬ 
nology  is  fully  integrated  with 
operational  hardware  and  soft¬ 
ware  systems.  Software  develop¬ 
ment  documentation  is  complete. 
All  functionality  tested  in  simul¬ 
ated  and  operational  scenarios. 

Published  documentation  and 
product  technology  refresh  build 
schedule.  Software  resource 
reserve  measured  and  tracked. 
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Table  C-3.  Software  TRL  Definitions,  Descriptions,  and  Supporting  Information 
(Source:  IT  TRL  Working  Group  Minutes,  November  9,  2004)  (Continued) 


TRL 

Definition 

Description 

Supporting  Information 

9 

Actual  system  proven 
through  successful  mis¬ 
sion-proven  operational 
capabilities 

Level  at  which  a  software  tech¬ 
nology  is  readily  repeatable  and 
reusable.  The  software  based  on 
the  technology  is  fully  integrated 
with  operational  hardware/soft¬ 
ware  systems.  All  software 
documentation  verified.  Suc¬ 
cessful  operational  experience. 
Sustaining  software  engineering 
support  in  place.  Actual  system. 

Production  configuration  man¬ 
agement  reports.  Technology 
integrated  into  a  reuse  “wizard”, 
out-year  funding  established  for 
support  activity. 

C.1.3  The  TRL  Concept  for  Manufacturing 

The  TRL  framework  should  also  be  used  to  assess  the  readiness  of  a  critical 
manufacturing  technology.  Similar  to  a  hardware  or  software  technology,  a  manufac¬ 
turing  technology  will  mature  through  TRLs  1,  2,  and  3.  However,  in  most  cases,  this 
maturation  will  occur  independently  of  a  specific  component  for  a  specific  system.  For  a 
manufacturing  technology  to  be  identified  as  critical  from  a  TRA  perspective,  it  must  be 
critical  in  the  program  context  of  cost,  schedule,  and  performance  as  described  in 
Appendix  D.  Therefore,  a  TRA  for  a  critical  manufacturing  technology  will  begin  with 
TRL  4,  where  the  associated  critical  performance  technology  will  have  been  validated  in 
a  laboratory  environment  at  the  component  and/or  breadboard  level. 

Table  C-4  shows  the  TRLs  used  to  assess  manufacturing  technologies.  The  manu¬ 
facturing  technology  may  be  related  to  any  combination  of  infrastructure,  materials,  pro¬ 
cesses  or  methods,  and  measurement. 

Table  C-4.  Manufacturing  Technology  TRL  Definitions, 

Descriptions,  and  Supporting  Information 
[Source:  Joint  Defense  Manufacturing  Technology  Panel  (JDMPT) 
Manufacturing  Readiness  Level  Subgroup ] 


TRL 

Definition 

Description 

Supporting  Information 

1 

Basic  principles 
observed  and 
reported. 

NA 

2 

Technology  con¬ 
cept  and/or  appli¬ 
cation  formulated. 

NA 

3 

Analytical  and 
experimental  criti¬ 
cal  function  and / 
or  characteristic 
proof  of  concept. 

NA 
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Table  C-4.  Manufacturing  Technology  TRL  Definitions, 
Descriptions,  and  Supporting  Information 
[Source:  Joint  Defense  Manufacturing  Technology  Panel  (JDMPT) 
Manufacturing  Readiness  Level  Subgroup ]  (Continued) 


TRL 

Definition 

Description 

Supporting  Information 

4 

Component  and / 
or  breadboard 
validation  in  a 
laboratory 
environment. 

The  new  technology  has 
been  demonstrated  in  a 
laboratory  environment  on 
simple  design  parts  using 
similar  types  of  materials 
that  would  be  used  in  the 
intended  application. 

This  is  the  lowest  level  of  production  readiness. 
Component  technologies  must  have  matured  to 
at  least  TRL  4.  At  this  point,  few  requirements 
have  been  validated,  and  there  will  be  a  large 
number  of  engineering/design  changes.  Com¬ 
ponent  physical  and  functional  interfaces  have 
not  been  defined.  Materials,  machines,  and 
tooling  have  been  demonstrated  in  a  laboratory 
environment.  Inspection  and  test  equipment 
have  been  demonstrated  in  a  laboratory  envi¬ 
ronment.  Manufacturing  cost  drivers  are 
identified.  Producibility  assessments  have  been 
initiated. 

5 

Component  and / 
or  breadboard 
validation  in  a 
relevant 
environment. 

The  new  technology  has 
been  demonstrated  in  a 
laboratory  environment  on 
design  parts  of  the  same 
level  of  complexity  and 
using  the  same  types  of 
materials  that  would  be 
used  in  the  intended 
application. 

Component  technologies  must  have  matured  to 
at  least  TRL  5.  At  this  point,  all  requirements 
have  not  been  validated,  and  there  will  be  sig¬ 
nificant  engineering/design  changes.  Compo¬ 
nent  physical  and  functional  interfaces  have  not 
been  defined.  Materials,  machines,  and  tooling 
have  been  demonstrated  in  a  relevant  manufac¬ 
turing  environment,  but  most  manufacturing 
processes  and  procedures  are  in  development 
(or  ManTech  initiatives  are  ongoing).  Inspection 
and  test  equipment  have  been  demonstrated  in 
a  laboratory  environment.  Production  cost 
drivers/goals  are  analyzed.  System-level  DTC 
goals  are  set.  Producibility  assessments  are 
ongoing. 

6 

System/subsys¬ 
tem  model  or 
prototype  demon¬ 
stration  in  a  rele¬ 
vant  environment. 

The  new  technology  has 
been  demonstrated  in  a  pre- 
production  environment  on 
design  parts  of  the  same 
level  of  complexity  and 
using  the  same  types  of 
materials  that  would  be 
used  in  the  intended  appli¬ 
cation.  Appropriate  quality 
levels  have  been  achieved. 

During  the  prototype  demonstration,  phase 
requirements  are  validated  and  defined. 

However,  there  will  still  be  many  engineering/ 
design  changes,  and  the  physical  and  functional 
interfaces  are  not  yet  fully  defined.  Component 
technologies  must  have  matured  to  at  least  TRL 

6.  Raw  materials  are  initially  demonstrated  in 
relevant  manufacturing  environment.  Similar 
processes  and  procedures  have  been  demon¬ 
strated  in  a  relevant  manufacturing  envi¬ 
ronment.  At  this  point,  there  are  likely  major 
investments  required  for  machines  and  tooling. 
Inspection  and  test  equipment  should  be  under 
development.  Producibility  assessments  are 
ongoing,  and  trade  studies  have  been  con¬ 
ducted.  A  production  Cost  Reduction  Plan  is 
developed.  Production  goals  are  set. 
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Table  3-4.  Manufacturing  Technology  TRL  Definitions, 
Descriptions,  and  Supporting  Information 
(Source:  JDMTP  Manufacturing  Readiness  Level  Subgroup)  (Continued) 


TRL 

Definition 

Description 

Supporting  Information 

7 

System  prototype 
demonstration  in 
an  operational 
environment. 

The  new  technology  has 
been  demonstrated  in  a 
relevant  production  envi¬ 
ronment  on  design  parts  of 
the  same  level  of  complexity 
and  using  the  same  types  of 
materials  that  would  be 
used  in  the  intended  appli¬ 
cation.  Appropriate  quality 
and  throughput  levels  have 
been  achieved. 

Component  technologies  must  have  matured  to 
at  least  TRL  7.  At  this  point,  engineering/design 
changes  should  decrease.  Physical  and 
functional  interfaces  should  be  clearly  defined. 

All  raw  materials  are  in  production  and  available 
to  meet  planned  LRIP  schedule.  Pilot  line 
manufacturing  processes  and  procedures  set 
up  and  under  test.  Processes  and  procedures 
are  not  yet  proven  or  under  control.  During  this 
phase,  initial  producibility  improvements  should 
be  underway.  DTC  estimates  are  less  than 

125  percent  of  goals.  Detailed  production 
estimates  are  established. 

8 

Actual  system 
completed  and 
qualified  through 
test  and 
demonstration. 

The  new  technology  has 
been  demonstrated  in  a  pilot 
production  environment  on 
production-representative 
parts  of  the  same  level  of 
complexity  and  using  the 
same  types  of  materials  that 
would  be  used  in  the 
intended  application.  Appro¬ 
priate  quality  and  throughput 
levels  have  been  achieved. 
Process  has  been  proven 
and  is  under  control  for 

LRIP. 

Component  technologies  must  have  matured  to 
at  least  TRL  8.  At  this  point,  engineering/design 
changes  should  decrease  significantly.  Physical 
and  functional  interfaces  should  be  clearly 
defined.  All  raw  materials  are  in  production  and 
available  to  meet  the  planned  LRIP  schedule. 
Manufacturing  processes  and  procedures  have 
been  proven  on  the  pilot  line,  are  under  control, 
and  are  ready  for  LRIP.  During  this  phase,  initial 
producibility  risk  assessments  should  be 
completed.  Production  cost  estimates  meet 

DTC  goals. 

9 

Actual  system 
proven  through 
successful  mis¬ 
sion  operations. 

The  new  technology  has 
been  demonstrated  in  an 

LRIP  environment  on 
intended  parts  and  using  the 
intended  types  of  materials. 
Process  has  been  proven 
and  under  control  for 
production. 

During  LRIP,  all  systems  engineering/design 
requirements  should  be  met  and  there  should 
only  be  minimal  system  engineering/design 
changes.  Technologies  must  have  matured  to 
at  least  TRL  9.  Materials  are  in  production  and 
available  to  meet  planned  production 
schedules.  Manufacturing  processes  and 
procedures  are  established  and  controlled  in 
production  to  three-sigma  or  some  other 
appropriate  quality  level.  Machines,  tooling  and 
inspection  and  test  equipment  deliver  three- 
sigma  or  some  other  appropriate  quality  level  in 
production.  Production  risk  monitoring  is 
ongoing.  LRIP  actual  costs  meet  estimates. 

C.2  ASSESSING  HARDWARE  CTEs 

Applying  the  TRL  definitions  to  assess  the  maturity  of  hardware  technologies 
appears  to  be  straightforward.  For  a  particular  technology,  the  level  of  technical  readiness 
that  best  describes  the  accomplishments  and  evidence  in  light  of  the  TRL  definitions 
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should  be  assigned.  In  practice,  this  approach  is  more  difficult  than  it  appears  because  the 
TRL  definitions  often  fail  to  account  for  all  real-life  situations. 

TRL  definitions  involve  several  dimensions.  One  could  be  called  the  application 
level,  which  assumes  values  of  device,  component,  subsystem,  system,  and  system  of 
systems.  Another  could  be  the  environment,  which  assumes  values  of  laboratory,  mathe¬ 
matical  model,  physical  simulation,  field  test,  and  operational  use.  Scale  and  performance 
levels  are  still  other  dimensions. 

Some  of  these  dimensions  are  used  explicitly  in  the  TRL  definitions,  and  some  are 
not.  In  any  event,  the  level  of  technical  readiness  is  determined  by  a  combination  of  these 
dimensions.  When  the  accomplishment  and  evidence  fail  to  match  the  definition,  the 
assessor  must  use  judgment  regarding  the  relevance  of  what  has  been  accomplished  and 
ask  whether  the  accomplishment  is  equivalent  to  the  TRL  definition. 

Of  these  dimensions,  environment  is  perhaps  the  most  difficult  to  interpret.  Both 
TRL  5  and  TRL  6  depend  on  demonstration  in  a  relevant  environment.  While  the  specif¬ 
ics  of  a  relevant  environment  depend  on  the  technology,  the  criterion  is  as  follows: 

A  relevant  environment  for  the  demonstration  of  a  technology  is  a  set  of 
test  conditions  that  provide  confidence  that  skillful  application  of  that 
technology  to  an  item  (component,  subsystem,  or  system)  will  support  the 
required  (threshold)  functionality  of  that  item  across  the  full  spectrum  of 
required  operational  employments. _ 


This  criterion  intentionally  avoids  the  word  “prove”  because  that  would  establish 
a  higher,  sometimes  unreasonable,  standard.  However,  the  need  to  support  the  full  range 
of  required  operational  employments  implies  that  one  or  a  few  demonstrations  conducted 
under  the  most  favorable  conditions  are  not  adequate.  If  a  body  of  data  or  accepted  theory 
supports  with  confidence  that  the  efficacy  of  a  technology,  though  demonstrated  only  in 
some  useful  environment,  can  be  extended  to  the  full  spectrum  of  employments,  the 
demonstration  can  be  considered  to  have  been  employed  in  a  relevant  environment. 


Demonstration  of  a  technology  in  a  relevant  environment  requires  suc¬ 
cessful  trial  testing  that  either 

(1)  shows  that  the  technology  satisfies  functional  need  across  the  full 
spectrum  of  operational  employments,  or 

(2)  shows  that  the  technology  satisfies  the  functional  need  for  some 

important  operational  employment  and  uses  accepted  techniques  to 
extend  confidence  over  all  required  operational  employments. _ 
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The  steps  or  activities  in  system  development  programs  differ  with  the  type  of 
system.  However,  some  of  the  steps  and  some  of  the  terminology  are  generally  applica¬ 
ble.  Table  C-5  lists  numerous  steps  typical  of  hardware  system  development  programs 
and  indicates  the  TRL  that  is  supported  by  this  accomplishment.  In  this  table,  “sup¬ 
ported”  means  that  the  step  is  at  least  partial  justification  for  assigning  the  indicated  TRL. 
“Tested”  means  not  just  that  a  test  was  run,  but  also  that  the  test  results  are  consistent 
with  the  needs  of  the  application.  Note  that  the  items  under  Accomplishment  usually 
include  an  application  level  and  an  environment  and  sometimes  include  a  scale  or  per¬ 
formance  level.  The  accomplishments  that  support  TRLs  of  4  through  7  are  of  particular 
relevance  to  TRAs  for  Milestone  B. 


Table  C-5.  Attainment  of  Technical  Readiness  for  Hardware  CTEs 


Accomplishment 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Discovery  of  physical  or  mathematical  principle 

X 

Characterization  of  the  principle 

X 

Application  envisioned  and  described 

X 

Concept  of  application  analyzed 

X 

Critical  functionality  empirically  confirmed 

X 

Proof  of  concept  demonstrated  in  laboratory 

X 

Scale-up  or  other  extension  as  needed  by  concept 

X 

X 

Breadboard  or  component  tested  in  laboratory 

X 

Producibility  and  cost  estimated 

X 

X 

Engineering  Development  Model  (EDM)47  of  component  tested  in 
laboratory 

X 

EDM  of  component  tested  in  relevant  environment 

X 

Prototype  component  integrated  into  a  system48  EDM 

X 

X 

System  EDM  tested  in  simulated  environment 

X 

System48  tested  in  limited  field  experiments 

X 

X 

System58  tested  in  relevant  environment51 

X 

47  A  pre -prototype  used  for  engineering  development,  functionally  the  near-equivalent  of  a  prototype  but 
differing  from  a  prototype  in  noncritical  features. 

System  or  subsystem. 

49  Either  EDM  or  prototype  at  the  system  or  subsystem  level. 

58  Prototype  or  high-fidelity  model  at  system  or  subsystem  level. 
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Table  C-5.  Attainment  of  Technical  Readiness  for  Hardware  CTEs  (Continued) 


Accomplishment 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

System  tested  in  operational  environment52 

X 

Production  system  tested  in  operational  environment 

X 

Production  system  proven  in  mission  operations 

X 

C.2.1  Aircraft 

Aircraft  are  likely  to  have  CTEs  in  aerodynamic  configuration  and  controls,  air¬ 
frame  structure  and  aeroelasticity,  flight  control  systems,  and  propulsion.  In  addition, 
rotary  wing  aircraft  have  CTEs  in  power  transfer,  rotor  hub,  and  blades.  CTEs  could  also 
be  factors  in  mission  equipment,  secondary  power,  environmental  control,  and  other  sys¬ 
tems,  depending  on  the  aircraft’s  missions.  A  variety  of  methods  and  facilities  are  used  to 
demonstrate  these  different  technologies. 

For  example,  demonstrations  such  as  analysis,  computational  fluid  dynamics 
(CFD)  investigations,  wind  tunnel  tests53,  and  flight  tests  are  normally  used  for  the  aero¬ 
dynamic  configuration  and  controls.  When  aerodynamic  configurations  indicate  large 
departures  from  existing  aircraft,  free-flight  models  (manned  or  unmanned)  are  some¬ 
times  used.  Similarly,  a  variety  of  methods  and  facilities  are  used  for  airframe,  flight 
control,  and  other  aeronautical  disciplines.  Table  C-6  shows  a  few  of  the  most  often  used 
means  to  demonstrate  aircraft  technologies  and  indicates  the  TRLs  that  can  be  supported 
by  these  demonstrations. 


Table  C-6.  Aircraft  Demonstrations  and  Supported  TRLs 


Demonstration 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Aerodynamic  Configuration  and  Controls 

Analysis  using  theory  and  data 

X 

X 

X 

Computational  fluid  dynamics 

X 

X 

Exploratory  wind  tunnel  tests 

X 

X 

5'  Meeting  the  most  significant  requirements. 

52  Proving  operational  requirement  can  be  met. 

53  Often  with  a  variety  of  scale  models  tested  in  several  different  wind  tunnels  to  obtain  data  for  different 
flight  conditions  and  mission  phases. 
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Table  C-6.  Aircraft  Demonstrations  and  Supported  TRLs  (Continued) 


Demonstration 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Wind  tunnel  tests  of  specific  configuration 

X 

X 

X 

Flight  control  wind  tunnel  tests 

X 

X 

Free  flight  of  scaled  aircraft 

X 

X 

Flight  tests  of  EDM54  aircraft 

X 

Airframe  Structure  and  Aeroelasticity 

Analysis 

X 

X 

X 

Finite  element  analysis 

X 

X 

Laboratory  tests  of  structural  elements 

X 

X 

X 

Loads  tests  of  major  structure  (e.g.,  wing)55 

X 

Flutter  tests  in  wind  tunnel 

X 

X 

Flight  tests  of  prototype  aircraft  to  g  limits 

X 

Flight  Control  System 

Analysis  (stability  margins,  control  authority) 

X 

X 

X 

6  Degrees  of  Freedom  (DOF)  simulations 

X 

X 

Laboratory  tests  of  sensors  and  actuators 

X 

X 

Flight  tests  with  surrogate  aircraft 

X 

X 

Flight  test  of  EDM  or  prototype  aircraft 

X 

Propulsion 

Hot  section  material  tests  in  laboratory 

X 

X 

Test  cell  testing  of  components55 

X 

X 

Test  cell  testing  of  engine  core 

X 

Wind  tunnel  tests  of  prototype  engine  and  inlet 

X 

X 

Flight  test  of  prototype  engine 

X 

Integrated  Aircraft 

Flight  tests  of  EDM  aircraft 

X 

Flight  tests,  EDM  with  mission  equipment 

X 

54  EDM  functionally  near  equivalent  to  prototype. 

55  These  tests  determine  the  strength  and  rigidity  of  the  major  structure. 

55  Blade  tests,  combustion  chamber  tests,  compressor  and  turbine  tests,  gear  box  and  power  transfer  tests, 
and  so  forth,  perhaps  conducted  with  surrogate  engines. 
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Table  C-6.  Aircraft  Demonstrations  and  Supported  TRLs  (Continued) 


Demonstration 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Flight  tests  of  complete  prototype  aircraft57 

X 

Qualification  tests  of  early  production  aircraft 

X 

Operational  use  of  production  aircraft 

X 

C.2.2  Ground  Vehicles 

Most  new  military  vehicle  concepts/systems  can  be  expected  to  involve  CTEs. 
Combat  and  tactical  vehicles  face  new  requirements  driven  by  new  threats  and  new  or 
extended  performance  needs  of  operational  forces.  Utility  and  general-purpose  vehicles, 
many  of  which  are  adapted  versions  of  commercial  vehicles,  also  can  be  required  to  pro¬ 
vide  special  performance  characteristics  that  exploit  new  technologies  or  novel  applica¬ 
tion  of  existing  technologies.  Table  C-7  is  a  sampling  of  military  vehicle  demonstrations 
and  indicates  the  TRLs  that  can  be  supported  by  these  demonstrations. 


Table  C-7.  Ground  Vehicle  Demonstrations  and  Supported  TRLs 


Demonstration 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Power  Package 

Analysis  using  theory  and  data 

X 

X 

X 

Computational  chemistry,  heat  transfer,  and  mechanics 

X 

X 

Laboratory  proof  of  principle  experiments 

X 

X 

Propulsion  component  small-scale  bench  tests 

X 

X 

Propulsion  package  scaled  test  stand  tests  simulating  rep¬ 
resentative  environments 

X 

X 

Full-scale  test  stand  testing  representative  of  operational 
environments 

X 

X 

Propulsion  package  proving  ground  tests  in  a  representa¬ 
tive  surrogate  vehicle 

X 

Armament  (Gun  and  Ammunition) 

Preliminary  concept  development  using  top-level  analysis, 
data  collection,  and  experience 

X 

X 

X 

57  Tests  throughout  flight  envelope  and  missions. 
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Table  C-7.  Ground  Vehicle  Demonstrations  and  Supported  TRLs  (Continued) 


Demonstration 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Thorough  computational  analysis 

X 

Propellant  chemistry 

X 

Interior  ballistics  for  gun  and  ammunition 

X 

Flight  dynamics 

X 

Warhead  and  penetrator  performance 

X 

Scaled  laboratory  tests  of  system  components 

X 

Full-scale  laboratory  tests  under  operationally  relevant 
conditions 

X 

X 

Full-scale  tests  of  gun  and  ammunition  integrated  with 
mount  and  recoil  system 

X 

Protection  (Passive  and  Reactive  Components) 

Materials  development 

X 

X 

X 

Armor  configuration  concept  development  and  preliminary 
analysis 

X 

X 

Computational  analysis  simulating  relevant  threats 

X 

X 

Scaled  laboratory  tests  of  generic  configurations  against 
simulated  relevant  threats 

X 

Full-scale  tests  of  generic  configurations  against  simulated 
relevant  threats 

X 

Full-scale  tests  of  combat  system  representative  configura¬ 
tions  against  relevant  threats. 

X 

The  automotive  features  of  any  class  of  military  vehicles  are  likely  to  exploit  criti¬ 
cal  technologies  in  propulsive  power,  drive  trains,  platform  stability,  suspension  systems, 
and  endurance.  Demonstration  of  critical  technology  efficacy  requires  various  means  of 
analysis,  test,  and  verification.  In  most  cases,  these  analyses  and  tests  are  unique  to  the 
military  environment. 

The  protection  requirements  and  features  of  combat  and  tactical  vehicles  are 
unique  aspects  driven  by  combat  environments.  CTEs  should  be  anticipated  in  vehicle 
integrated  passive  protection  against  diverse  weapon  and  munitions  threats.  Similarly,  as 
threats  increase  and  become  more  sophisticated,  CTEs  appear  that  have  reactive  (e.g., 
explosive  armor)  or  active  (e.g.,  detection  and  attack  of  threat  munitions)  aspects. 
Evaluation  of  the  maturity  of  these  technologies  is  often  made  by  developing  extensions 
to  existing  analysis  and  test  capabilities. 
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Battlefield  dominance  continually  demands  that  weaponry  improve  to  meet 
growing  and  changing  threat  capabilities.  Technology  improvement,  development,  and 
exploitation  are  the  principal  means  to  meet  the  demand.  Most  new  weapons  involve  a 
gun  or  a  missile  and  all  their  associated  technologies.  Assessing  the  maturity  of  these 
technologies  requires  strong  analysis,  test,  and  demonstration. 

C.2.3  Missiles  and  Guided  Weapons 

The  development  program  for  a  missile  or  other  guided  weapon  is  quite  different 
from  that  of  a  “platform”  vehicle,  and  the  program  for  a  solid  propellant  rocket  is  differ¬ 
ent  from  that  of  a  liquid  propellant  rocket.  Most  military  missiles  have  structure,  propul¬ 
sion,  guidance,  flight  control,  and  payload.  Each  of  these  comprises  numerous  elements 
that  must  function  together  to  meet  the  objectives  of  the  system,  and  any  of  these  ele¬ 
ments  can  depend  upon  CTEs.  To  assess  the  maturity  of  these  technologies,  issues  that 
should  be  considered  in  performance  demonstrations  include  how  the  environments  com¬ 
pare  with  the  environments  of  the  intended  uses  and  how  the  performance  exposes  what 
is  required. 

Missile  structural  integrity  and  flight  control  are  highly  interdependent.  Structural 
bending  modes,  placement  of  accelerometers,  control  system  time  constants,  aerody¬ 
namic  loads  and  control  moments,  and  reaction  controls  must  all  work  together  to  achieve 
stable,  controlled  flight.  Structural  rigidity  and  inertial  properties  can  first  be  computed 
during  computer-aided  design  (CAD)  and  confirmed  by  ground  tests.  Aerodynamics  can 
be  determined  by  analysis  and  wind  tunnel  tests.  High-fidelity,  6  DOF  simulations  can 
represent  the  complete  missile  in  its  intended  flight  environment.  Components  that  are 
tested  in  hardware-in-the-loop  (HWIL)  simulations  can  reasonably  be  considered  to  be 
TRL  4.  Assuming  that  flight  accelerations  and  vibrations  are  important  to  the  functioning 
of  a  component  and  then  testing  that  component  while  it  is  carried  on  a  surrogate  missile 
could  achieve  TRL  5.  After  the  components  are  integrated  into  an  EDM  dynamically 
correct  missile  and  flown,  perhaps  on  a  flight  with  pre-programmed  maneuvers,  the  com¬ 
ponents  can  be  considered  TRL  6  if  the  environment  is  relevant  for  those  components. 

Missile  guidance  systems  can  include  a  variety  of  sensor  types.  Several  types  of 
test  environments  have  been  found  to  be  useful  for  particular  types  of  sensors.  These 
include  anechoic  chambers  for  radars  and  other  radio  frequency  (RF)  systems,  terrain 
tables  for  visual  and  infrared  (IR)  detectors,  towers  overlooking  tactical  targets,  captive 
carry  on  aircraft  and  missiles,  and  free  flight.  The  maturity  associated  with  these  sensors 
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depends  on  the  fidelity  of  the  relevant  features  of  the  environment  and  on  the  fidelity  of 
the  test  article  when  compared  with  the  final  product.  If  a  tower  can  provide  the  correct 
viewpoint  and  range  to  a  target  and  if  motion  is  not  important,  perhaps  a  tower  test  of  a 
prototype  sensor  can  be  adequate  to  assess  TRL  5.  However,  if  motion  is  important,  a 
captive  carry  test  might  be  necessary  to  achieve  TRL  5.  Since  motion  is  almost  always 
important  to  missile  guidance  systems,  captive  carry  for  TRL  5  and  demonstration  on  a 
prototype  or  surrogate  missile  for  TRL  6  are  suggested  as  the  norms. 

The  missile  payload  will  usually  be  a  warhead,  but  it  could  be  a  sensor  or  a  com¬ 
munications  system.  The  warhead  requires  a  safe  and  arm  function  and  a  fuze  function, 
and  these  functions  might  be  performed  by  separate  devices,  possibly  integrated  with  the 
guidance  system  or  a  central  processor.  The  warhead  fill  (the  explosive  material)  can  be 
characterized  in  a  laboratory— first  in  small  samples  and  then  in  larger  samples.  Numer¬ 
ous  tests  are  required  for  effectiveness  and  safety  to  bring  the  fill  to  TRL  3.  If,  as  is  usu¬ 
ally  the  case,  the  fill  has  been  used  successfully  in  other  warheads,  it  might  be  considered 
TRL  6  or  7  depending  upon  the  similarity  of  the  present  application’s  environment  to  that 
of  the  earlier  applications.  Prototype  cases  loaded  with  the  intended  fill  should  be  tested 
for  application-specific  requirements  (drop  tests,  projectile-impact  tests,  cook-off  tests, 
fragment  pattern,  and  so  forth)  to  reach  TRL  5.  The  devices  providing  the  safe  and  arm 
and  fuse  functions  should  follow  a  similar  development,  finally  being  integrated  with  a 
prototype  warhead  and  flown  on  a  missile  that  can  create  a  near-operational  environment 
to  demonstrate  TRL  6. 

For  all  these  subsystems,  the  developer  should  ensure  that  none  of  the  potential 
critical  items  are  overlooked.  To  illustrate  this  point,  the  missile  propulsion  subsystem 
will  be  considered  in  more  detail.  Table  C-8  shows  some  typical  development  activities 
for  solid-fuel  propulsion  systems  (solid  rockets)  and  the  TRL  that  can  be  supported  by 
these  activities.  A  rocket  of  this  type  has  propellant  that  must  meet  requirements  for 
energy,  burn  rate,  strength,  stability,  and  safety,  among  others.  The  propellant  in  a  rocket, 
the  “grain,”  must  be  supported  so  that  it  does  not  separate  from  the  motor  case  under 
shock  or  high  acceleration.  The  case  usually  must  be  insulated  from  the  grain  to  preserve 
the  strength  of  the  case  material,  and,  of  course,  the  nozzle  must  not  erode  excessively 
during  motor  firing.  The  table  lists  important  activities  for  determining  the  maturity  of  the 
technologies  needed  for  these  functions.  “Test”  normally  means  a  series  of  trials  and  not 
a  single  trial. 
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Table  C-8.  Typical  Steps  in  Development  of  a  Rocket  Motor  and  the  TRLs  Supported 


Accomplishment 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Grain 

Discovery  of  an  energetic  material 

X 

Laboratory  synthesis  of  sample  formulations 

X 

X 

Chemical  properties  of  propellant  established58 

X 

X 

Physical  properties  of  propellant  established59 

X 

X 

Test  of  grain  in  laboratory  motor  surrogate 

X 

Grain  Support  and  Case  Insulation 

Static  test  of  heavy  case  motor59 

X 

Shock  and  vibration  tests 

X 

Sled  test  of  heavy  weight  motor 

X 

X 

Case 

Case  material  properties  established51 

X 

Trial  cases  built 

X 

X 

Hydro  burst  test  of  prototype  case 

X 

Nozzle 

Erosion  tests  of  nozzle  materials 

X 

Nozzle  gimbal  test 

X 

X 

Erosion  test  of  prototype  nozzle 

X 

Motor 

Static  test  of  prototype  motor 

X 

Flight  test  of  prototype  motor52 

X 

X 

Flight  test  of  prototype  motor  on  prototype  missile 

X 

Flight  test  of  production  motor  on  objective  missile58 

X 

Operational  use  of  motor  in  missile  system 

X 

58  For  example,  specific  energy,  bum  rate,  sensitivity,  chemical  stability. 

59  For  example,  density,  strength,  ductility,  rheology. 

59  Having  grain  support  and  insulation  of  objective  motor. 

51  Strength  at  temperature,  ductility,  machinability,  weld  strength,  and  so  forth. 

52  On  dynamic  surrogate  of  objective  missile  or  on  developmental  flight  test  of  objective  missile. 
55  Demonstrating  full  operational  envelope. 
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For  liquid  fuel  rockets,  different  items  are  important.  There  must  be  movement 
and  metering  of  fuel  and  oxidizer.  There  might  be  throttling  or  multiple  starts.  There 
might  be  cooling  of  the  nozzle  with  fuel.  Relevant  conditions  can  include  very  low  ambi¬ 
ent  pressures  and  longitudinal  and  lateral  accelerations  that  can  be  achieved  only  in  flight. 

Air-breathing  rockets  have  the  additional  needs  to  establish  inlet  performance  and 
flammability  limits  over  a  wide  range  of  Mach  numbers  and  ambient  pressures.  Demon¬ 
strations  can  include  connected  tests  (inlet  connected  to  an  air  source)  and  free-flow  tests 
including  inlet,  captive  carry,  and  free-flight  tests.  These  could  merit  TRLs  of  4,  5,  5,  and 
6,  respectively,  if  the  test  articles  of  the  free-flight  tests  are  functionally  representative 
prototypes. 

C.2.4  Ships  and  Ship  Systems 

Ships  are  likely  to  have  CTEs  in  hydrodynamic  hull  form,  materials  and  struc¬ 
tures,  propulsion,  drag  reduction,  and  motion  controls.  Ship  systems  such  as  sensors 
(radar/sonar),  weapons  (torpedoes/missiles),  hotel  (waste  disposal/desalination/material 
movement),  and  aircraft  interfaces  (elevators)  will  require  some  additional  CTEs.  Ships 
also  have  CTEs  related  to  survivability,  such  as  signatures,  countermeasures,  and  intact 
and  damaged  stability.  A  wide  variety  of  methods  and  facilities  are  used  to  demonstrate 
these  different  technologies. 

Ships  are  usually  large  and  complex;  therefore,  prototyping  of  a  complete  system, 
such  as  a  new  hull  form,  is  expensive  and  time  consuming.  The  types  of  demonstrations 
used  normally  for  ship  hull-form  technologies  include  analysis,  CFD  investigations, 
towing  tank  model  scale  tests,  and  land-based  subsystem  tests.  For  ship  configurations 
that  represent  large  departures  from  the  existing  base  of  knowledge,  large-scale  proto¬ 
types  are  sometimes  used.  For  example,  submarine  signature  improvements  are  always 
evaluated  at  least  at  the  1/4  scale  in  water. 

Similarly,  a  variety  of  methods  and  facilities  are  used  for  structures  and  materials, 
motion  control,  and  other  ship-related  disciplines.  Table  C-9  shows  a  few  of  the  most 
often  used  means  to  demonstrate  ship  and  ship  systems  technologies  and  indicates  the 
TRLs  that  can  be  supported  by  these  demonstrations.  For  ship-based  missile  systems,  see 
Section  C.2.3.  Torpedo  development  would  follow  an  approach  similar  to  that  of  a  mis¬ 
sile  system.  The  ship  signatures,  hydrodynamics,  and  hull  form  development  usually  run 
in  parallel,  so  the  demonstrations  are  linked  as  indicated  in  Table  C-9.  The  technologies 
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Table  C-9.  Ship-Related  Demonstrations  and  Supported  TRLs 


Demonstration 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Hull  Form,  Hydrodynamics,  and  Signatures 

Analysis  using  theory  and  data 

X 

X 

X 

Computational  hydroacoustics  and  hydrodynamics 

X 

X 

Exploratory  tow  tans  and  water  tunnel  tests 

X 

X 

Model  scale  tests  of  specific  configurations 

X 

X 

Reanalysis  using  empirical  data 

X 

X 

1 /4-scale  tests  in  a  realistic  environment 

X 

X 

Final  full-scale  predictions 

X 

Ship  Materials  and  Structures 

Initial  theoretical  analysis 

X 

X 

Finite  element  analysis 

X 

X 

Stress  tests  of  structural  elements 

X 

X 

X 

Structural  model  tests  in  a  tow  tank 

X 

X 

At-sea  material  evaluation 

X 

X 

Large-scale  evaluation  in  a  seaway 

Motion  Control  System 

Stability  and  control  analysis 

X 

X 

X 

6  DOF  simulations 

X 

X 

Subsystem  tests  of  sensors  and  actuators 

X 

X 

Free-running  model  scale  tests 

X 

X 

Prototype  evaluations 

X 

Propulsion 

Concept  development  and  analysis 

X 

X 

Proof-of-concept  tow  tank  component  tests 

X 

X 

Prime  mover  land-based  tests 

X 

Integrated  propulsion  scale  model  tests 

X 

X 

Full-scale  prediction  of  propulsion  efficiency 

X 

Ship  Sensors  and  Hotel  Systems 

Concept  development  in  a  laboratory 

Component  evaluations 

X 

X 

Prototype  development  and  testing 

X 

X 

Evaluation  on  an  existing  ship 

X 

X 
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Table  C-9.  Ship-Related  Demonstrations  and  Supported  TRLs  (Continued) 


Demonstration 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Integrated  Ship  Concept 

Integration  of  system  into  ship  concept 

X 

Integrated  prototype  at-sea  testing 

X 

Production  ship  first-of-class  trials 

X 

Production  ship  in  mission  environment 

X 

of  active  drag  reduction  are  treated  similar  to  those  of  a  propulsion  subsystem,  such  as  a 
new  propeller,  and  would  follow  the  propulsion  approach.  Passive  drag  reduction  sys¬ 
tems,  such  as  hull  shaping,  are  treated  similar  to  the  hull  form  development  approach. 

C.2.5  Hardware  for  IT  Applications 

The  two  examples  presented  describe  the  approach  for  assessing  the  technical 
readiness  of  hardware  CTEs  used  in  IT  applications. 

C.2.5.1  Effective  Information  Displays 

Infantry  soldiers  on  the  battlefield  oper¬ 
ate  in  an  extremely  demanding  environment. 

While  soldiers  are  expected  to  carry  the  equiva¬ 
lent  of  a  laptop  computer,  the  form  and  fit  of  a 
conventional  laptop  is  awkward.  This  CTE 
example  is  concerned  with  the  display  technol¬ 
ogy  of  an  integrated  computer  system  having  an 
ergonomic  fit  and  form  for  use  by  infantry 
soldiers. 

A  high-tech  monocle  [based  on  Microelectromechanical  Systems  (MEMS)  tech¬ 
nology]  to  project  images  directly  onto  the  retina  has  been  selected.64  The  military  has 
tested  early  prototypes  of  this  technology.  Commanders  of  Stryker  vehicles  have  the 


for  Soldiers  on  the  Battlefield 


64  Such  a  system  is  expected  to  be  more  rugged  than  conventional  approaches,  to  be  able  to  be  read  in  the 
daylight,  and  to  have  higher  resolution  than  a  conventional  display.  Furthermore,  because  essentially 
all  the  light  generated  enters  the  eye,  the  device  is  extremely  energy  efficient  and  thereby  reduces 
demand  on  the  local  power  supply. 
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option  of  viewing  the  onboard  battlefield  computer  with  a  helmet-mounted  display 
(HMD).  Another  prototype  system  has  experimented  with  this  technology  to  increase 
situational  awareness  by  providing  helicopter  pilots  a  digital  display  of  the  battlespace. 

The  experience  gained  from  testing  the  display  with  soldiers  in  Stryker  vehicles 
and  helicopter  pilots  provides  a  technical  readiness  of  no  higher  than  TRL  6  based  on 
evidence  from  these  field  trials.  The  operational  environment  of  the  infantry  soldier  is 
quite  different  from  the  two  tested  applications.  Achieving  a  TRL  7  or  higher  would 
require  testing  the  display  in  the  operational  environment  of  the  infantry  solider. 

C.2.5.2  Data  Centers  for  Medical  Records 

The  complete  medical  records  for  members  of  the  Services  will  be  located  in  a 
distributed  computer  database  and  be  readily  accessible  on  demand  throughout  the  world. 
Owing  in  part  to  the  size  of  images  generated  by  diagnostic  instruments,  this  database 
will  quickly  grow  in  size  to  become  one  of  the  largest  DoD  databases.  As  a  consequence, 
the  data  centers  hosting  these  data  will  push  the  limits  for  storage  size  and  bandwidth  for 
data  transport  within  the  data  center.  This  example  compares  two  technologies:  the  use  of 
an  interconnect  technology  expected  to  become  the  default  choice  for  high-performance 
systems  and  a  conventional,  established  technology. 

Data  centers  consisting  of  a  cluster  of  servers,  persistent  storage,  and  networked 
connections  to  clients  often  use  many  network  and  transport  technologies  for  intercon¬ 
necting  these  elements.  Replacing  these  multiple  networking  technologies  with  a  unified 
interconnection  technology,  such  as  InfiniBand,  is  desirable  for  satisfying  this  range  of 
requirements.65  The  technology  is  available  and  has  been  used  previously  to  build  a 
robust,  high-capacity  interconnection  fabric  for  a  data  center  with  similar  requirements. 
Sufficient  evidence  exists  for  a  TRL  of  7  or  higher. 

Conventional  10-gigabit  Ethernet  is  a  mature  alternative  that  has  had  significant 
historical  success.  Unfortunately,  Ethernet’s  performance  is  severely  limited  because  it 
presently  does  not  have  the  quality  of  service  (QoS)  and  fault  tolerance  found  in 
Infiniband. 

A  dimension  of  “relevant  environment”  includes,  in  this  case,  the  capability  to 
maintain  and  expand  the  data  center.  This  will  present  a  problem  if  the  interconnect 


65  Having  multiple  networking  technologies  at  the  level  of  box-to-box  connection  increases  system 
complexity  and  reduces  reliability. 


C-24 


technology  fails  to  meet  market  acceptance  expectations  and,  consequently,  disappears 
from  the  marketplace.  In  that  case,  a  TRL  of  7  would  no  longer  be  appropriate  for 
Infiniband,  whose  market  acceptance,  to  date,  is  not  assured.  Often,  no  clear  choices  exist 
in  these  situations  between  balancing  the  old  and  proven  approaches  (though  of  limited 
capability)  with  emerging  approaches  (whose  long-term  survivability  has  yet  to  be 
established).  The  best  practice  is  to  expose  the  issues  and  evaluate  conservatively. 

C.3  ASSESSING  SOFTWARE  CTEs 

As  in  hardware  systems,  the  definitions  of  TRLs  as  applied  to  software  involve 
several  dimensions.  At  the  application  level  are  values  of  device,  component,  subsystem, 
system,  and  system  of  systems  for  hardware  and  algorithms,  software  components,  soft¬ 
ware  programs,  and  software  packages  for  software.  Another  dimension,  discussed  at 
length  in  Appendix  D,  includes  environment  (or  application)  with  values  of  integration 
laboratory,  user  environment,  logical  relationships,  data  environment,  and  possibly 
interfaces.  Other  system-wide  dimensions  include  obsolescence,  scalability,  and  through¬ 
put  and  are  usually  expressed  in  terms  of  system-wide  requirements,  but  the  hardware 
components  often  contributes  to  meeting  these  requirements.  As  in  the  hardware  TRLs, 
some  of  these  terms  are  used  explicitly  in  the  TRL  definitions,  and  some  are  not.  The 
combination  of  these  dimensions  determines  any  TRL.  When  the  accomplishment  and  the 
definition  do  not  match,  the  assessor  must  use  his/her  judgment  regarding  the  relevance 
of  what  has  been  accomplished  and  ask  whether  the  accomplishment  is  equivalent  to  the 
TRL  definition. 

In  assessing  software’s  technical  readiness,  one  must  be  aware  of  the  proper  use 
of  the  terms  “relevant  environment”  and  “operational  environment.”  Claiming  technical 
readiness  in  a  relevant  environment  (TRL  7  or  higher)  requires  a  detailed  architecture  that 
fully  exposes  all  components  and  elements  affecting  the  operation  of  the  critical  software 
element.  Claiming  technical  readiness  in  an  operational  environment  (TRL  6  or  higher) 
requires  evidence  of  the  acceptable  performance  of  the  software  element  under  opera¬ 
tional  factors,  including,  for  example,  system  loading,  user  interaction,  and  realistic  com¬ 
munications  environment  (e.g.,  bandwidth,  latency,  jitter).  In  other  words,  claiming  a 
TRL  5  or  higher  requires  a  detailed  architecture,  and  claiming  a  TRL  7  or  higher  requires, 
in  addition  to  the  detailed  architecture,  defining  the  operational  environment  and  evi¬ 
dence  of  acceptable  performance  in  the  operational  environment. 
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In  practice,  assessing  the  maturity  of  the  software  components  of  IT  systems  can 
be  difficult.  Table  C-10,  adapted  from  Table  C-5  for  the  hardware  technology,  can  be 
used  as  a  guide  of  the  type  of  data  needed  in  support  a  given  maturity  level.  This  table 
lists  the  numerous  steps  typical  of  software  system  development  programs  and  indicates 
the  TRLs  that  can  be  supported  by  these  accomplishments. 


Table  C-10.  Attainment  of  Technical  Readiness  for  Software  CTEs 


Accomplishment 

TRL  Supported 

1 

2 

3 

4 

5 

6 

7 

8 

9 

Discovery  of  mathematical  principle  or  algorithm 

X 

Characterization  of  the  principle 

X 

Application  envisioned  and  described 

X 

Concept  of  application  analyzed 

X 

Critical  functionality  empirically  confirmed  and  implemented 
software 

X 

Proof  of  concept  demonstrated  in  simulation 

X 

Scale-up  or  other  extension  as  needed  by  concept 

X 

X 

Component  tested  in  simulation 

X 

Producibility  and  cost  estimated 

X 

X 

Software  component  tested  in  an  integration  laboratory 

X 

Software  component  tested  in  a  relevant  environment 

X 

Prototype  component  integrated  into  a  system  prototype 

X 

X 

System  tested  in  a  simulated  environment 

X 

System  tested  in  a  limited  field  experiments 

X 

X 

System  tested  in  a  relevant  environment 

X 

System  tested  in  an  operational  environment 

X 

Production  system  tested  in  an  operational  environment 

X 

Production  system  proven  in  mission  operations 

X 

Brief  examples  estimating  the  level  of  technical  readiness  for  software  elements 

follow. 

C.3.1  Information  Integration  of  Unstructured  Data  (See  Figure  C-l) 

This  situation  highlights  CTE  assessment  considerations  in  programs  that  inter¬ 
face  with  many  semi-autonomous  organizations  at  the  information,  data,  and  processing 
level  but  have  little  or  no  design  influence  within  the  organizations  beyond  the  interface. 
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Figure  C-1.  Three  Dimensions  of  Information  Integration 
(Source:  A.D.  Jhingran,  N.  Mattos,  and  H.  Pirahesh,  “Information  Integration: 

A  Research  Agenda,”  IBM  Systems  Journal,  Vol.  41 ,  No.  4,  2002) 

In  such  as  system,  extensible  Markup  Languuage  (XML)  can  be  used  to  access  struc¬ 
tured  and  unstructured  data.66  XML  would  describe  unstructured  data  through  XML 
schemas,  and  data  access  would  be  provided  via  XQuery  and  XPath  standards. 

If  the  application  were  a  mission  planning  system,  several  DoD-unique  concerns 
would  have  to  be  considered: 

•  Because  of  the  limited  control  over  design  and  operation  internal  to  the  orga¬ 
nization  hosting  the  data  sources,  an  increased  emphasis  is  placed  on  the 


66  The  data  in  a  structured  data  source  are  strongly  typed,  and  relationships  are  described  by  a  schema. 
The  data  are  organized  in  tables  and  accessed  via  a  relational  database.  Structured  Query  Language 
(SQL)  is  supported  for  accessing  information  in  the  database.  Unstructured  data  consists  of  practically 
everything  else,  including  documents,  images,  data  sets,  field  reports,  and  maps.  While  some  of  these 
unstructured  data  types  are  semi-structured,  which  can  be  exploited  for  organization  and  accessibility, 
these  heterogeneous  data  sets  have  begun  to  be  unified  only  recently.  A  query  should  transparently 
combine  data  from  relational  tables,  the  XML  database,  and  data  retrieved  from  external  servers. 
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inter-organization  interface  for  delineating  areas  of  responsibility  (i.e.,  func¬ 
tional  allocation)  and  standards  for  representing  data  using  XML. 

•  The  system  needs  to  accommodate  the  restrictive  nature  of  highly  classified 
data  sources  while  providing  access  to  less  classified  and  most  likely  unclas¬ 
sified  sources.  For  this  system  to  be  useful,  the  security  model,  along  with  its 
implementation,  must  successfully  provide  access  while  enforcing  security 
policies  in  a  manner  that  still  allows  for  automated  and  efficient  operation. 

•  Although  base  standards  have  been  issued  for  XQuery  and  XPath,  it  is  not 
clear  that  they  have  achieved  sufficient  maturity  for  this  application. 

CTEs  would  be  found  in  the  XML  data  models  and  their  interaction  with  XQuery; 
in  the  interface  definitions,  including  functional  allocation  among  the  organization;  and  in 
the  implementation  of  security  policy.  Without  any  documented,  relevant  DoD  experi¬ 
ence,  a  TRL  of  4  is  the  highest  level  that  should  be  assigned. 

C.3.2  Distributed  Resource  Sharing  (See  Figure  C-2) 

This  example  discusses  CTEs  associated  with  the  capability  to  process,  interpret, 
and  distribute  an  unprecedented  quantity  of  data  collected  from  sensor  networks,  over¬ 
head  assets,  and  other  technical  collection  means  in  a  timely  manner— a  netcentric  war¬ 
fare  scenario.  The  technical  approach  is  to  implement  a  grid  service  architecture  that  is 
currently  being  developed  in  a  consortium  environment  for  coordinated  resource  sharing 
and  problem  solving  in  a  dynamic,  multiorganizational  setting.67 

CTEs  are  mostly  confined  to  the  suitability  and  performance  of  the  architecture  in 
a  military  environment.  Specifically,  concern  involves  accommodating  DoD  security 
policy  and  performance  over  a  network  of  limited  bandwidth,  including  response  to  unex¬ 
pected  events  that  causes  resources  to  disappear  temporarily  (e.g.,  severance  of  a  commu¬ 
nications  link). 


67  Storage,  computational,  and  communication  resources  will  be  shared  by  providing  standard,  open,  and 
general-purpose  protocols  and  interfaces  for  capabilities,  including  authentication,  authorization, 
resource  discovery,  and  resource  access.  This  includes  direct  and  managed  access  to  sensors,  pro¬ 
cessors,  software,  communications  bandwidth,  storage,  file  systems,  database,  and  servers.  These 
resources  can  be  used  collectively  on  existing  standard  Web  service  components  in  a  coordinated 
fashion  to  deliver  negotiated  qualities  of  service,  relating  for  example  to  response  time,  throughput, 
availability,  and  security.  The  thrust  is  to  provide  a  capability  for  dynamically  establishing  resource¬ 
sharing  arrangements  with  any  interested  member  and  thus  create  something  more  than  a  plethora  of 
balkanized,  incompatible,  noninteroperable  distributed  systems. 
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Figure  C-2.  Elements  Involved  in  Resource  Sharing 
(Source:  Ian  Foster,  “The  Grid:  A  New  Infrastructure  for  21st  Century  Science,” 
Physics  Today,  Vol.  55,  Issue  2,  2000) 

A  highly  promoted  way  of  developing  software  and  standards  is  via  a  consortium 
with  wide  participation  from  commercial,  governmental,  and  academic  organizations. 
This  is  becoming  an  accepted  approach  in  the  software  and  communications  sectors  to 
promote  open  standards  and  accommodate  user  needs  better.  In  the  present  example,  grid 
technology  has  undergone  continuous  development  for  more  than  10  years  and  has 
resulted  in  several  standards  and  released  software  packages.  Through  active  participa¬ 
tion,  the  program  intends  to  use  the  standards  as  they  currently  exist  and  influence  their 
evolution  to  accommodate  currently  unsatisfied  needs. 

Because  the  selected  architecture  has  only  established  its  viability  in  primarily 
scientific  and  limited  commercial  domains,  a  TRL  of  no  higher  than  4  should  be 
assigned.  Achieving  a  higher  level  of  technical  readiness  is  possible  only  in  the  context  of 
a  detailed  architecture  and  within  a  distributed  military  environment.  For  example 
achieving  the  required  QoS  level  is  critical  to  the  viability  of  this  system.  QoS  is  diffi¬ 
cult— if  not  impossible— to  assess  accurately  without  an  operational  system.  The  diffi¬ 
culty  in  assessing  QoS  arises  because  QoS  degrades  as  a  system  is  stressed  from 
workload,  dynamic  reconfiguration,  and  component  failures. 
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C.3.3  Autonomic  Computing68  (See  Figure  C-3) 
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Figure  C-3.  Evolving  to  Autonomic  Operations 
(Source:  IBM  Global  Services  and  Autonomic  Computing,  IBM  White  Paper,  October  2002) 

Dependence  on  IT  systems  during  critical  tactical  operations  places  exceedingly 
high  requirements  on  their  reliability,  availability,  and  security.  A  new  strategy  for 
increasing  IT  system  reliability  and  availability  while  at  the  same  time,  reducing  depen¬ 
dence  on  human  intervention,  incorporates  an  autonomic  system  to  manage  system 
operation  dynamically.69 

Most  of  the  technology  required  to  build  automatic  systems  either  does  not  exist 
or  exists  at  a  research  level/early  prototype  stage.  Procurement  of  a  fully  autonomic  sys¬ 
tem  is  not  technically  viable  at  present.  A  TRL  of  3  is  the  maximum  assessment. 

In  the  larger  context  of  a  well-defined,  incremental  approach  for  achieving  a  fully 
autonomic  capability,  technology  selection  and  evaluation  should  be  focused  on  the 


68  Much  as  the  autonomic  nervous  system  in  humans  frees  our  conscious  brain  from  the  burden  of 
controlling  low-level  but  vital  functions  and  coping  with  deviations  from  normal  operation  (e.g., 
infection),  an  autonomic  system  as  part  of  an  IT  system  makes  the  IT  system  self-managing.  That  is, 
the  system  become  self-configuring,  self-healing,  self-optimizing,  and  self-protecting  with  minimal 
human  intervention. 

69  An  autonomic  system  is  implemented  as  a  collection  of  interacting,  automatically  managed  elements. 
These  elements  include  hardware  resources  (e.g.,  storage,  processing,  or  communications)  or  software 
resources  (e.g.,  application  program,  database,  or  operating  system)  or  even  other  automatically 
managed  IT  systems.  Each  autonomic  element  is  responsible  for  managing  its  own  internal  state  and 
behavior.  Through  interacting  with  other  automatically  managed  elements  and  the  external  world,  the 
state  of  the  system  is  driven  toward  consistency  with  the  given  goals. 
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capabilities  required  for  the  current  increment.70  The  current  strategy  calls  for  evolving 
the  system  though  five  increments  (basic,  managed,  predictive,  adaptive,  autonomic)  that 
progress  from  manually  managed  to  automatically  managed. 

As  an  example,  consider  a  program  undergoing  development  of  its  second  incre¬ 
ment,  which  is  focused  on  consolidation  and  presentation  of  state  and  performance  data 
through  management  tools.71  The  software  technology  for  functions  of  consolidation  and 
presentation  is  available  and  has  been  demonstrated  to  operate  in  a  relevant  operational 
environment.  Hence,  the  evidence  will  likely  support  a  TRL  of  5  or  higher. 

C.3.4  Radio  Frequency  Identification  (RFID)  Tags  for  Material  Assets 
Management 

Management  of  military  supplies  and  equipment  is  exceedingly  complex  because 
current  inventory  accounting  systems  are  outdated,  have  limited  interoperability,  and  are 
implemented  with  poorly  documented  software.  Knowing  the  status  of  these  material 
assets  (e.g.,  current  location,  expected  date  of  delivery  for  new  assets,  condition,  and 
ownership)  reduces  costs  and  improves  capability. 

RFID  tags  provide  automatic  identification  of  tagged  assets  as  they  pass  through 
locations  equipped  with  interrogators.  The  military  has  used  selective  RF  tagging  of  large 
or  expensive  items  for  many  years.  However,  as  spurred  by  commercial  organizations 
such  as  Wal-Mart  in  managing  their  supply  chain,  RFID  tagging  will  reach  the  point 
where  it  is  technically  and  economically  possible  to  tag  practically  all  levels  of  material 
objects.  Furthermore,  not  only  will  the  tags  identify  the  object  type,  but  they  can  also 
encode  item-specific  information  such  as  expiration  date  and  lot  number. 

DoD  will  probably  be  in  a  position  to  use  a  commercially  proven  technology  with 
an  inherently  low  technical  risk.  While  this  will  certainly  be  true  for  several  common 
technology  issues  (e.g.,  the  cost  of  tags  and  good  readability  by  fixed  interrogators), 
military  IT  systems  for  collecting,  processing,  and  using  RFID  tags  are  expected  to  face 
many  technical  challenges.  For  example, 

•  It  is  common  to  want  to  know  where  the  object  is  and  not  where  it  was  when 
the  RFID  tag  was  last  read.  Knowing  an  object’s  whereabouts  requires  the 


70  It  is  the  designer's  responsibility  to  select  and  develop  technologies  that  naturally  build  toward  future 
increments  and  this  is  not  a  consideration  of  technical  readiness  for  the  current  increment. 

71  The  first  increment  defined  and  collected  the  data  that  is  being  consolidated. 
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integration  of  tag  information  with  the  in-transit  visibility  (ITV)  server.  Other 
asset  management  systems  that  will  need  to  interoperate  with  the  RFID  sys¬ 
tem  include  the  Government  Freight  Management  (GFM)  system,  the  Global 
Air  Transportation  Execution  System  (GATES),  the  Surface  Transportation 
Management  System  (STMS),  and  the  Movement  Tracking  System  (MTS). 

•  If  objects  are  tagged  at  multiple  levels  (e.g.,  the  item  itself,  a  box  of  items,  a 
pallet  of  boxes  of  items,  a  shipping  container  housing  the  pallet,  and  so 
forth),  not  all  tags  will  necessarily  be  interrogated  at  the  same  time.  As  the 
contents  of  shipping  containers  get  rearranged  and  distributed  and  the  pallets 
get  broken  down,  mechanisms  and  procedures  must  be  put  in  place  to  deter¬ 
mine  the  whereabouts  of  the  material  assets. 

•  RFID  only  works  when  interrogators  are  in  place  to  read  the  tags.  Since 
deployment  destinations  are  not  always  known  in  advance,  either  interroga¬ 
tors  must  be  in  place  and  operational  before  tagged  assets  are  moved  or  a 
way  to  accommodate  a  loss  of  contact  must  be  developed. 

While  these  problems  do  not  appear  to  require  new  technology  as  part  of  their 
solution,  they  do  require  a  careful  consideration  of  interactions,  interoperability  with 
other  systems,  and  sensible  use  of  the  RFID  capability.  Until  systems  have  been  devel¬ 
oped  and  real-world  experience  has  been  gained,  a  TRL  of  5  or  less  is  appropriate. 

In  addition,  RFID  tagging  presents  other  technical  challenges.  DoD  will  poten¬ 
tially  use  RFID  tags  and  receptors  in  extreme  environmental  conditions  that  most  com¬ 
mercial  firms  do  not  face.  Potential  wireless  security  concerns  also  exist  if  sensitive 
material  is  being  tracked.  These  issues  require  new  technology  as  part  of  their  solution, 
and,  therefore,  a  TRF  of  4  is  more  appropriate. 

C.4  ASSESSING  MANUFACTURING  CTEs 

The  readiness  of  a  manufacturing  technology  is  evaluated  in  context  of  under¬ 
standing  the  risks  associated  with  the  industrial  process  and  then  developing  and  imple¬ 
menting  risk  mitigation  plans.  The  risk  elements  can  be  classified  over  the  following 
areas: 

•  Technology  and  industrial 

•  System  design 

•  Materials 

•  Cost  and  funding 

•  Process  capability  and  control 


C-32 


•  Quality  management 

•  Personnel  skills  and  availability 

•  Facility  capability  and  capacity 

•  Manufacturing  planning,  scheduling,  and  control. 

Table  C-ll  indicates  accomplishments  that  should  occur  consistent  with  these 
threads  for  a  manufacturing  technology  to  achieve  a  given  TRL.  The  following  examples 
demonstrate  these  concepts. 


Table  C-11.  Attainment  of  Technical  Readiness  for  Manufacturing  CTEs 


Accomplishment 

TRL  Supported 

4 

5 

6 

7 

8 

9 

Emerging  breadboard  design  options  provide  insight  into  potential 
manufacturing  problems  with  the  industrial  base  infrastructure  (facilities 
and  manpower),  materials,  methods,  and  measurement  (inspection 
and  test  equipment) 

X 

Breadboard  design  options  provide  insight  needed  to  validate  charac¬ 
teristics  and  potential  geometries 

X 

Various  strategies  are  identified  to  mitigate  technical  and  cost  risk 

X 

Prototype  brassboard  design  has  actual  components,  subsystems,  or 
systems  that  have  associated  manufacturing  processes,  materials,  and 
methods 

X 

Preliminary  assessment  of  manufacturing  assembly  sequences 

X 

Industrial  base  infrastructure  (facilities  and  manpower)  capabilities 
along  with  measuring  and  test  equipment  initially  are  evaluated 

X 

Cost  accounted  for  on  high-risk  manufacturing  areas  and  plans  are 
developed  to  mitigate  risk 

X 

Quality  management  model  understood 

X 

Manufacturing  processes,  materials,  and  assembly  methods  have 
been  developed  for  a  production  environment— ideally  in  a  pre-produc¬ 
tion  facility  or  better 

X 

Design  maturing;  key  materials  and  process  characteristics  have  been 
identified,  and  planning  is  taking  place  for  managing  (process  control 
as  appropriate) 

X 

Detailed  manufacturing  risk  assessment  covering  industrial  base  infra¬ 
structure  (facilities  and  manpower),  materials  (availability,  producibility 
characteristics),  methods  (mature  processes),  measurement  (inspec¬ 
tion  and  test  equipment),  and  costs 

X 

Quality  management  structure  is  identified 

X 
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Table  C-11.  Attainment  of  Technical  Readiness  for  Manufacturing  CTEs  (Continued) 


Accomplishment 

TRL  Supported 

4 

5 

6 

7 

8 

9 

Appropriate  throughput  levels  have  been  achieved 

X 

Initial  goals  set  for  yields,  quality,  and  reliability 

X 

Manufacturing  processes,  materials,  and  assembly  methods  have 
been  demonstrated  on  production-representative  articles,  with  no 
known  significant  manufacturing  risk 

X 

Yields,  quality,  reliability  and  cost  are  within  25  percent  of  goals 

X 

Design  mature.  Process  requirements  are  proven  and  validated 

X 

Quality  management  structures  are  in  place 

X 

Manufacturing  processes  are  efficient  and  acceptable  in  factory 
environment 

X 

Design  producible.  Used  to  produce  production  articles  for  IOT&E  and 
the  field 

X 

Design-to-cost  (DTC)  and  production  goals  are  met 

X 

C.4.1  Example:  Manufacturing  Technology  for  Laser  Diode  Arrays 

Laser  diode  arrays  are  critical  components  for  high-efficiency  and  reliable  solid- 
state  lasers  that  are  used  in  laser  rangefinders  and  laser  designators.  Manufacturing  tech¬ 
nologies  were  pursued  to  reduce  the  manufacturing  cost  by  a  factor  of  two.  Major  steps  in 
the  manufacturing  process  are  as  follows: 

•  Epitaxial  growth  of  laser  structure  on  gallium  arsenide  (GaAs)  substrates 

•  Wafer  processing  to  define  individual  elements  through  photolithography  and 
metallization  processes 

•  Wafer  cleaving  to  produce  1  cm  x  1  mm  rectangular  bar  elements 

•  Optical  coating  on  bar  facets  to  produce  high-reflectivity  mirror  on  one  edge 
and  partially  reflecting  mirror  on  opposite  edge 

•  Bond/solder  bars  in  heat  sink  package 

•  Test  the  completed  laser  diode  array  for  specified  performance. 


TRL 

Example  Descriptions 

4 

Manufacturing  concepts  identified  from  producibiiity  studies:  Initial  studies  indicated  that 
the  cost  drivers  in  bar  production  are  wafer  processing  and  touch  labor  in  cleaving  and  coating 
bars.  An  increase  in  wafer  size  will  increase  throughput  and  reduce  processing  cost  per  bar.  The 
automation  of  bar  stacking  in  coating  fixtures  will  reduce  touch  labor  and  improve  yields.  New 
package  materials  and  assembly  techniques  are  expected  to  increase  laser  diode  array  reliability. 
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TRL 

Example  Descriptions 

5 

Strategies  identified  to  mitigate  technical  and  cost  risk:  Change  from  2-in.  to  4-in.  wafers  for 
Molecular  Beam  Epitaxy  (MBE)  growth.  Identify  machines  to  automate  bar  stacking  in  coating 
fixtures.  Preload  bars  in  bonding  fixtures  to  reduce  solder  thickness  in  laser  diode  array  package 
and  improve  reliability  and  thermal  performance. 

MBE  Machine  for  Fabrication  of  Laser  Diode  Wafers 


4-in.,  3-in.,  and  2-in.  Wafers  Shown  for  Comparison 
(Compared  Against  the  Size  of  a  Quarter) 


TRL 

Example  Descriptions 

6 

Manufacturing  process  developed  and  applied:  Initial  4  in. -wafer  growth  and  evaluation  of 
laser  performance  and  wavelength  uniformity.  The  yield  of  specification  lasers  is  dependent  on 
the  uniformity  of  material  and  device  wavelength.  Design,  build,  and  evaluate  automated  bar 
stacking  machines  for  function  and  yield  of  good  devices.  Develop  bar  bonding  techniques  to 
improve  yield  and  the  reliability  of  devices.  Demonstrate  4-in.  wafer  MBE  growth  meeting  center 
wavelength  and  wavelength  variation  specifications  across  full  wafer.  Complete  automated  bar 
stacking  machine  and  demonstrate  improved  yield  in  test  lots.  Demonstrate  array  package  with 
reduced  solder  thickness  process  and  evaluate  thermal  performance  and  reliability. 
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Maps  Showing  Wavelength  Uniformity 
Initial  4-In.  Wafer  Growth 


Machine  To  Stack  Bars  In  Coating 
Fixtures  To  Reduce  Touch  Labor 


TRL 

Example  Descriptions 

7 

Prototype  pre-manufacturing  system:  Initial  evaluation  of  individual  manufacturing  processes  for 
growth  and  fabrication  of  wafers.  Evaluate  automated  bar  stacking,  coating,  and  unstacking  steps  for 
quantity  of  devices.  Initial  evaluation  of  producing  new  laser  diode  array  packages  including  perform¬ 
ance  tests. 

8 

Manufacturing  process  maturity  demonstration:  Pilot  line  demonstration  of  manufacture  of  laser 
diode  bars  from  4-in.  wafers  and  using  automated  bar  handling  machines.  Determine  yield  for  each 
process.  Determine  yield  of  laser  diode  array  packages  that  meet  specifications  for  power  and 
wavelength. 

9 

Manufacturing  processes  proven:  Laser  diode  bars  manufactured  in  quantity  with  target  yields 
and  cost.  Laser  diode  array  packages  are  manufactured  in  quantity,  with  required  yields  for  specified 
performance  and  proven  reliability. 

Unmounted  807-nm  Laser  Diode  Bars 


8-Bar  Laser  Diode  Array 
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C.4.2  Example:  Metal  Joining  Technologies  [High-Strength  Low-Alloy 
(HSLA)-IOO  Steel] 

The  importance  manufacturing  technology  for  metal  joining,  especially  welding, 
was  driven  home  on  April  10,  1963,  with  the  sudden  and  catastrophic  sinking  of  the  USS 
Thresher  (SSN-593)  off  the  New  Hampshire  coast,  killing  all  aboard.  As  technologies 
and  capabilities  improved,  the  demand  on  submarines  (i.e.,  deeper,  faster,  and  quieter) 
grew.  Similar  demands  were  also  placed  on  surface  ships.  These  performance  improve¬ 
ments  required  new  materials  and  processes  that  were  often  a  variation  on  existing  mate¬ 
rials  and  processes.  However,  even  minor  changes  to  a  material’s  properties  or  a 
manufacturing  process  should  require  extensive  manufacturing  proofing  before  produc¬ 
tion,  as  was  the  case  with  HSLA-100  steel. 

C.4.2.1  TRL  4 

What  Occurred:  For  the  trial  plate 
production  phase  of  the  HSLA-100  steel  pro¬ 
ject,  an  initial  150-ton  production  of  HSLA- 
100  steel  was  melted  and  rolled  by  Phoenix 
Steel  Corporation  in  1986  to  the  interim 
specification.  This  process  used  a  conven¬ 
tional  electric  furnace  and  ingot  casting  prac¬ 
tice,  conducted  to  achieve  a  very  low  carbon 
composition.  The  minimum  strength  and  toughness  requirements  of  the  interim  specifi¬ 
cation  were  met  in  the  initial  production  of  HSLA-100  steel  plates  in  gages  from  1/4  to 
2  in.  Optimum  properties  in  HSLA-100  steel  plate  resulted  from  aging  temperatures  from 
1,150  to  1,275  °F.  Upon  receipt  of  the  HSLA-100  steel  plate  from  the  trial  productions, 
an  review  commenced  to  evaluate  HSLA-100  steel  plate  and  welding  using  the  processes 
and  procedures  for  High  Yield  Strength  (HY)-100  steel  ship  and  submarine  structural 
applications— but  with  reduced  heat  or  no  preheat.  The  evaluation  of  HSLA-100  steel 
plate  properties  and  welding  demonstrated  that  HSLA-100  steel  met  the  mechanical 
property  requirements  of  HY-100  steel  and  was  able  to  be  welded,  with  reduced  preheat 
requirements  and  using  the  same  welding  consumables  as  those  for  HY-100  steel  fabri¬ 
cation.  When  compared  with  HY-100  steel,  the  tensile  and  impact  toughness  properties  of 
the  plates  met  or  exceeded  the  requirements. 

Manufacturing  Perspective:  Trial  productions  were  run— but  on  a  limited  basis 
and  mainly  to  determine  if  performance  could  be  met.  For  example,  the  initial  production 


CRACK  EXTENSION  (mm) 
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of  HSLA-100  steel  was  rolled  by  the  Phoenix  Steel  Corporation.  However,  from  a 
manufacturing  perspective,  Phoenix  Steel’s  success  did  not  mean  that  other  manufac¬ 
turers  would  also  be  successful.  First,  other  manufacturers  would  have  used  different 
people  who  had  slightly  different  skills.  Second,  even  though  these  manufacturers  may 
have  used  the  same  machines,  these  machines  would  have  been  calibrated  and  set  differ¬ 
ently.  Third,  the  specification  was  interim,  indicating  that  it  was  not  fully  proven  or 
tested.  This  is  a  significant  manufacturing  concern.  Finally,  the  manufacturing  was 
migrating  from  a  process  that  required  pre-heating  to  processes  that  required  little  or  no 
pre-heating.  While  the  initial  findings  were  positive,  these  findings  did  not  indicate  that 
they  could  have  been  replicated  easily  by  other  production  houses.  The  major  issue  is 
this:  Just  because  an  item  can  be  produced  in  limited  quantities  or  in  a  lab  environment, 
the  transition  to  production  will  not  necessarily  be  low  risk. 

C.4.2.2  TRL  5 

What  Occurred:  Lukens 
Steel  Company  produced  a  second 
melt  of  HSLA-100  steel,  again  by 
electric  furnace  and  ingot  casting. 

Most  of  the  plate  produced  from 
the  heat  was  greater  than  2  in. 
thick,  primarily  for  ballistic  resis¬ 
tance  evaluation.  The  minimum 
strength  and  toughness  require¬ 
ments  were  met  in  plate  thick¬ 
nesses  that  ranged  from  1/2  to 
3-3/4  in.  A  double  austenitization 
and  quench  process  was  used  for  an  HSLA-100  steel  plate  in  gages  over  1-1/4  in.  to 
refine  the  heavy-plate  grain  structure  for  optimum  toughness.  The  HSLA-100  steel  was 
the  primary  material  used  in  the  certification  program,  from  the  production  to  the  interim 
specification.  The  certification  evaluation  included  continued  characterization  of  produc¬ 
tion  the  HSLA-100  steel  plate’s  mechanical,  physical,  and  fracture  properties.  However, 
the  main  focus  was  the  evaluation  of  weldability  and  welding  process  limits  for  structures 
of  high  restraint,  studies  of  fatigue  properties,  and  effects  of  marine  environments  on 
HSLA-100  steel.  The  results  of  low-cycle  fatigue  crack  initiation  tests  of  HSLA-100  steel 
and  weldments  and  high-cycle  fatigue  tests  in  air  and  seawater  indicated  properties 
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equivalent  to  HY-100  steel  in  every  case.  The  steels  showed  similar  fatigue  crack  growth 
rate  properties.  General  corrosion,  crevice  corrosion,  galvanic  corrosion,  and  high-veloc¬ 
ity  seawater  parallel  flow  and  cavitation  tests  of  HSLA-100  in  seawater  showed  that  the 
corrosion  behavior  of  HY-100  and  HSLA-100  steels  was  comparable. 

Manufacturing  Perspective:  With  advanced  lab  testing/development,  the  per¬ 
formance  capabilities  continued  to  mature;  however,  these  performance  capabilities  dif¬ 
fer.  For  example,  the  second  melt  of  HSLA-100  steel  was  by  Lukens  Steel  Corporation. 
They  have  a  different  factory  floor  plan  and  different  processes  than  those  of  Phoenix 
Steel.  Again,  product  evaluations  continued,  but  no  analysis  of  the  industrial  base  was 
conducted.  No  producibility  analysis  was  conducted  to  identify  potential  manufacturing 
process  improvements  that  would  lower  cost  and  risk.  As  the  specifications  emerged, 
they  probably  did  not  identify  the  material  and  process  tolerances  and  the  key  character¬ 
istics  of  those  processes  because  they  were  not  put  under  statistical  process  control  to 
ensure  uniform  quality. 


C.4.2.3  TRL  6 

What  Occurred:  The  evaluation  of 
HSLA-100  steel  production  plates  concluded 
that  the  mechanical  properties  of  production 
plate,  welding  and  weldability  screening  tests, 
fatigue  properties,  and  corrosion  properties 
demonstrated  that  the  system  was  viable  for  cer¬ 
tification  for  combatant  ship  structure.  System 
evaluation  by  explosion  bulge  and  crack-starter 
bulge  tests,  fragment  penetration  resistance 
tests,  and  ballistic  property  tests  were  success¬ 
fully  conducted.  In  1987,  the  Naval  Sea  Sys¬ 
tems  Command  (NAVSEA)  initiated  projects  at 
General  Dynamics  Electric  Boat  and  Newport  News  Shipbuilding  (NNS)  to  evaluate  the 
weldability  of  HSLA-100  steel  under  various  preheat  conditions  in  a  production  environ¬ 
ment.  The  results  of  the  weldability  evaluation  demonstrated  that  HSLA-100  steel  could 
be  welded  at  up  to  1. 25-in.  thick  at  60  °F  minimum  preheat,  with  the  same  processes  and 
consumables  being  used  for  HY-80/100  steels.  Ballistic  evaluations  demonstrated  that 
HSLA-100  steel  was  equivalent  to  HY-100  steel  and  weldments  in  ballistic  resistance. 
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Both  steels  were  comparable  to  Army  Rolled  Homogeneous  Armor.  In  March  1989, 
NAVSEA  certified  HSLA-100  for  surface  ship  construction  in  thicknesses  up  to  4  in. 

Manufacturing  Perspective:  With  prototyping  in  a  relative  environment,  the 
analysis  and  characterization  of  materials  and  processes  are  expanded.  For  example,  the 
prototyping  included  testing  of  both  Gas  Metal  Arc  Welding  (GMAW)  and  Submerged 
Arc  Welding  (SAW)  welding  of  HSLA-100,  and  from  a  thickness  of  1  in.  to  1  5/8  in.  and 
then  up  to  a  thickness  of  4  in.  Also,  the  prototype  evaluations  were  taking  place  in  multi¬ 
ple  industrial  facilities.  Each  of  these  facilities  had  different  personnel,  slightly  different 
processes,  and  different  vendors  of  materials  and  consumables.  As  they  were  prototyping, 
they  were  validating  their  processes  and  capturing  that  information  for  incorporation  into 
Military  Specifications  (MIL-SPECS).  The  need  for  capturing  quality  and  manufacturing 
planning  and  the  related  costs  to  validate  against  cost  targets  is  implied. 


C.4.2.4  TRL  7 


What  Occurred:  The  fabrication  of 
a  series  of  structural  performance  models 
was  completed  under  shipyard  welding  con¬ 
ditions.  Holding  bulkhead  panel  models, 
foundation  models,  and  a  full-scale  founda¬ 
tion  were  evaluated  and  demonstrated  satis¬ 
factory  structural  performance.  Electric  Boat 
fabricated  the  full-scale  foundation  and  a 
small,  heavy-gage  tank  model.  NNS  partially  completed  the  fabrication  of  a  full-scale 
hard  tank;  however,  a  funding  shortage  precluded  tests.  In  these  shipyard  fabrication 
exercises,  all  weld  cracking  was  related  to  Shielded  Metal  Arc  Welding  (SMAW)  and 
SAW  consumables  (where  cracking  occurred  even  when  HY-100  preheat  temperatures 
were  used)  or  to  improper  welding  practices.  No  heat- affected  zone  (HAZ)  cracking 
occurred  in  HSLA-100.  Hydrostatic  tests  of  full-gage  bulkhead  panel  models  are  an 
extreme  test  of  plating-to- stiffener  strength  and  HAZ  ductility.  The  HSLA-100  panel 
models  exceeded  anticipated  holding  pressure  levels,  withstanding  over  twice  the  holding 
pressure  of  identical  HY-100  panel  models.  A  series  of  foundation  beam  elements  (full- 
scale)  and  the  full-scale  SSN  688-type  AC  foundation  were  installed  and  tested  on  a 
floating  shock  platform.  The  structures  were  subjected  to  a  series  of  underwater  explo¬ 
sion  (UNDEX)  shock  tests.  For  a  series  of  3  UNDEX  events,  the  structural  response  of 
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the  HSLA-100  items  indicated  no  cracking  or  excessive  deformation  in  any  structural 
joint. 

Manufacturing  Perspective:  With  the  onset  of  prototyping  in  an  operational 
environment  and  with  subsystem  components,  the  actual  or  planned  manufacturing 
environment  is  mirrored.  Notice  that  production  testing  continued  at  both  Electric  Boat 
and  NNS  and  on  some  full-scale  parts.  The  one  red  flag  is  the  lack  of  full  testing  at  NNS 
because  of  funding  shortages,  especially  when  some  manufacturing  problems  still  had  to 
be  isolated  and  resolved  (e.g.,  the  cracking  caused  by  small  differences  in  consumables 
from  different  vendors).  This  indicates  that  key  characteristics  were  still  not  fully  identi¬ 
fied  because  there  were  interactions  between  material  factors  causing  the  cracking.  The 
improper  welding  practice  indicates  that  manufacturing  had  not  gotten  control  of  those 
processes  or  did  not  adequately  identify  those  process  steps  or  were  not  controlling  them 
from  a  quality  perspective. 

C.4.2.5  TRL  8 

What  Occurred:  In 

1989,  NAVSEA  certified 
HSLA-100  steel  for  surface 
ship  construction  in  thicknesses 
up  to  4  in.  At  that  time,  the 
USS  John  C.  Stennis  (CVN  74) 
was  approved,  indicating  that 
HSLA-100  steel  was  a  quali¬ 
fied  substitute  for  HY-80/100 
steel  in  CVN  construction.  The 
experience  base  for  welding 
HSLA-100  steel  was  too  limited  to  allow  the  wholesale  substitution  for  all  HY-80/100 
steel  in  the  unrestricted  areas  of  the  carrier.  Therefore,  an  implementation  plan  for  incor¬ 
porating  the  HSLA-100  steel  was  submitted,  and  NAVSEA  approved  this  plan.  NNS 
used  HSLA-100  steel  during  CVN  74  construction.  Approximately  700  tons  of 
HSLA-100  steel  plate  in  7/8-  and  1-in.  thicknesses  were  used  for  main  deck  panel  assem¬ 
blies  with  longitudinal  and  transverse  stiffeners  without  preheat  (65  to  80  °L  shop  tem¬ 
perature).  One  hundred  percent  magnetic  particle  inspection  was  performed  on  all  HSLA- 
100  butt  welds.  In  1,400  feet  of  7/8-in.  thick  HSLA-100  butt  weld  inspected  by  Magnetic 
Particle  Testing  (MT),  only  2  repairs  (8  in.  total)  were  required  (not  related  to  hydrogen- 
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type  defects).  The  same  length  of  1-in.  thick  HSLA-100  butt  weld  inspected  by  MT 
showed  no  defects.  A  total  of  1,250  tons  of  HSLA-100  were  used  in  CVN  74,  with  over 
4,000  ft  of  weldment  inspection  requiring  32  in.  total  repair  (less  than  0.01  percent).  The 
flight  deck  of  the  USS  Bataan  (LHD  5)  was  successfully  fabricated  with  HSLA-100  plate 
(in  place  of  HY-100  steel)  for  cost  savings,  as  were  subsequent  vessels  of  the  same  class. 

Manufacturing  Perspective:  With  technology  being  demonstrated  in  an  opera¬ 
tional  environment  (i.e.,  HSLA-100  welding  at  NNS  on  CVN  74),  the  technology  and 
manufacturing  processes  are  mature  enough  to  transition.  The  Navy  was  transitioning 
slowly  because  of  the  still  many  unknowns  related  to  large-scale  welding  efforts.  Thus, 
initial  production  was  limited  to  the  deck  area  only  and  to  1-in.  thick  plate.  As  production 
and  testing  progressed,  quality  data  came  in  and  further  supported  production  increases  to 
larger  thicknesses  and  other  areas  below  the  deck.  At  this  point,  yield  data  should  support 
the  use  of  selected  processes,  and  the  processes  should  be  stable  enough  to  allow  others 
to  replicate  the  results. 

C.4.2.6  TRL  9 

What  Occurred:  Because  of  the  experience  gained  on  CVN  74,  wholesale 
changes  to  HSLA-100  steel 
were  made  on  CVN  75. 

Approximately,  10,500  long 
tons  (LTs)  of  HSLA-100  steel 
were  inserted  into  CVN  75. 

Most  of  the  replacement  was 
for  decks  and  bulkheads  and 
some  built-up  stiffeners.  The 
HSLA-100  stiffeners  were 
short  spans  with  heavy  web/flange  members.  HSLA-100  steel  was  selected  to  replace 
HY-100  steel  for  fabrication  cost  reduction,  and,  as  a  consequence,  HSLA-100  steel  has 
been  used  in  place  of  HY-100  steel  in  the  construction  of  USS  John  C.  Stennis  (CVN  74), 
USS  Harry  S.  Truman  (CVN  75),  and  USS  Ronald  Reagan  (CVN  76).  On  the  CVN  76, 
NAVSEA  08  approved  the  substitution  of  HSLA-100  steel  for  HY-80/100  steel  structures 
outside  the  primary  shield  tank,  opening  another  area  for  substitution.  On  CVN  77, 
expended  use  of  HSLA-100  steel  plate  continues.  NNS  expects  to  qualify  reduced 
preheat  for  welding  up  to  2  in.,  adding  over  4,000  LT  of  HSLA-100  steel  where 
significant  fabrication  cost  reduction  is  gained  over  HY-100  steel  in  this  thickness  range. 


n 
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Depending  on  the  complexity  of  the  structure,  estimated  cost  savings  for  HSLA-100  steel 
vs.  HY-100  steel  fabrication  in  CVN  74  construction  range  from  $500  to  $3,000  per  ton 
of  fabricated  structure. 

Manufacturing  Perspective:  With  the  transition  from  low  rate  production  (LRP) 
to  full-scale  production  (FSP),  manufacturing  and  quality  processes  should  be  well 
documented,  and  efforts  should  be  put  into  place  to  improve  quality  and  productivity. 
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ACRONYMS  AND  ABBREVIATIONS  FOR  APPENDIX  C 


CAD 

computer-aided  design 

CFD 

computational  fluid  dynamics 

COTS 

commercial  off-the-shelf 

CTE 

Critical  Technology  Element 

CVN 

Carrier  Vessel  Nuclear 

DoD 

Department  of  Defense 

DOF 

degrees  of  freedom 

DTC 

design-to-cost 

EDM 

Engineering  Development  Model 

FDA 

Food  and  Drug  Administration 

FSP 

full-scale  production 

GaAs 

gallium  arsenide 

GATES 

Global  Air  Transportation  Execution  System 

GFM 

Government  Freight  Management 

GMAW 

Gas  Metal  Arc  Welding 

GOTS 

government  off-the-shelf 

HAZ 

heat- affected  zone 

HMD 

helmet-mounted  display 

HSLA 

High-Strength  Low-Alloy 

HWIL 

hardware-in-the-loop 

HY 

High  Yield  Strength 

IR 

infrared 

IT 

Information  Technology 

ITV 

in-transit  visibility 

JDMPT 

Joint  Defense  Manufacturing  Technology  Panel 

LHD 

Amphibious  Assault  Ship 

LRIP 

Low  Rate  Initial  Production 

LRP 

low  rate  production 

LT 

long  ton 
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MBE 

Molecular  Beam  Epitaxy 

MEMS 

Microelectromechanical  Systems 

MIL-SPECS 

Military  Specifications 

MT 

Magnetic  Particle  Testing 

MTS 

Movement  Tracking  System 

NASA 

National  Aeronautics  and  Space  Administration 

NAVSEA 

Naval  Sea  Systems  Command 

NNS 

Newport  News  Shipbuilding 

OT&E 

operational  test  and  evaluation 

QoS 

quality  of  service 

R&D 

research  and  development 

RF 

radio  frequency 

RFID 

radio  frequency  identification 

SAW 

Submerged  Arc  Welding 

Sim/Stim 

Simulation/Stimulation 

SMAW 

Shielded  Metal  Arc  Welding 

SQL 

Structured  Query  Language 

SSN 

Attack  Submarine  (Nuclear  Propulsion) 

STMS 

Surface  Transportation  Management  System 

TRA 

Technology  Readiness  Assessment 

TRL 

Technology  Readiness  Level 

UNDEX 

underwater  explosion 

XML 

extensible  Markup  Languuage 
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D.l  INTRODUCTION 


CTE  Defined 

A  technology  element  is  “critical”  if  the  system  being  acquired  depends  on 
this  technology  element  to  meet  operational  requirements  with  acceptable 
development  cost  and  schedule  and  with  acceptable  production  and 
operation  costs  and  if  the  technology  element  or  its  application  is  either 
new  or  novel.  Said  another  way,  an  element  that  is  new  or  novel  or  being 
used  in  a  new  or  novel  way  is  critical  if  it  is  necessary  to  achieve  the  suc¬ 
cessful  development  of  a  system,  its  acquisition,  or  its  operational  utility. 

Disciplined  identification  of  CTEs  is  important  to  a  program.  If  a  CTE  is  over¬ 
looked  and  not  brought  to  the  requisite  maturity  level  for  exploitation  at  the  start  of  Sys¬ 
tem  Design  and  Development  (SDD),  the  system  performance,  program  schedule,  and 
cost  could  be  jeopardized.  On  the  other  hand,  if  an  overly  conservative  approach  is  taken 
and  a  plethora  of  technologies  are  categorized  as  critical,  energy  and  resources  are  likely 
to  be  diverted  from  the  few  technologies  that  deserve  an  intense  maturation  effort.  If  a 
disciplined  process  with  due  diligence  does  lead  to  an  inordinate  number  of  CTEs,  this 
should  be  an  indication  that  the  proposed  development  is  reaching  too  far  for  its  goals. 

CTE  identification  begins  in  the  early  stages  of  systems  acquisition.72  Although 
final  identification  of  CTEs  is  not  expected  before  the  Concept  Decision,  the  team  devel¬ 
oping  the  Initial  Capabilities  Document  (ICD)  should  include  people  who  have  technical 
and  technology  backgrounds  to  ensure  that  materiel  elements  for  the  needed  capabilities 
are  plausible.  Restricting  the  capabilities  to  those  likely  to  be  achievable  will  prove  bene¬ 
ficial  for  any  program  that  intends  to  exploit  advanced  technology. 

A  major  part  of  the  CTE  identi¬ 
fication  process  should  occur  during 
Concept  Refinement.  The  Technology 
Development  Strategy  (TDS),  a  pro¬ 
duct  of  the  Concept  Refinement  phase, 
should  reflect  the  result  of  a  process 
sufficiently  thorough  and  disciplined  to 
identify  those  technologies,  including  CTEs,  that  have  a  realistic  potential  to  be  improved 
beneficially  in  the  Technology  Development  phase  and  exploited  in  the  SDD  phase. 


Best  Practice 

CTE  Identification  should  be  a 
continuing  element  of  every  pro¬ 
gram.  An  initial  determination  of 
CTEs  should  be  completed  during 
Concept  Refinement. 


72  See  Section  2  for  an  overview  of  the  systems  acquisition  process. 
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Failure  to  recognize  the  CTEs  at  this  stage  will  result  in  wasting  resources— time,  money, 
facilities,  and  so  forth— and  could  result  in  an  unfavorable  Milestone  B  decision. 

As  system  development  proceeds,  the  likelihood  exists,  through  necessity  or 
opportunity,  for  exploitation  of  technologies  not  previously  considered.  These  technolo¬ 
gies  deserve  full  consideration  to  decide  whether  they  are  critical  and  whether  they  are 
mature  enough  to  be  included  in  the  detailed  system  design. 

The  original  Department  of  Defense  (DoD)  Technology  Readiness  Level  (TRL) 
definitions  and  supporting  information  (see  Section  3,  Table  3-1  of  this  Deskbook)  were 
developed  primarily  with  performance-related  hardware  technologies  in  mind.  In  identi¬ 
fying  CTEs  and  assessing  their  maturity,  the  distinction  between  hardware  and  software 
technologies  became  important  because  different,  but  related,  procedures  and  metrics  are 
used  to  identify  and  assess  the  maturity  of  hardware  and  software  CTEs.  The  original  set 
of  definitions  suited  hardware  technologies  but  was  inadequate  for  software  technologies. 

Another  shortcoming  of  the  original  set  of  definitions  was  distinguishing  between 
performance-related  technologies  and  technologies  for  affordable  production.  The  CTE 
definition  includes  the  phrases  “with  acceptable  development  cost  and  schedule  and  with 
acceptable  production  and  operation  costs.”  Thus,  a  technology  that  “does  the  job”  but  is 
not  affordable  is  an  unacceptable  technology.  It  may  be  that  a  manufacturing  technology 
will  provide  the  required  affordability,  in  which  case  it  should  be  identified  as  a  CTE  if  it 
is  “new  or  novel.” 

The  following  sections  of  this  appendix  provide  suggestions  about  how  to  identify 
CTEs— hardware,  software,  and  manufacturing— for  a  variety  of  systems.73  These 
discussions  apply  equally  to  Major  Defense  Acquisition  Programs  (MDAPs)  and  Major 
Automated  Information  System  (MAIS)  programs.  Section  D.2  discusses  system  engi¬ 
neering  as  the  program  context  for  identifying  CTEs,  Section  D.3  covers  procedures  and 
practices  for  CTE  identification,  and  Section  D.4  contains  representative  questions/ 
inquiries  to  use  when  making  a  detailed  examination  of  a  system  to  identify  CTEs. 


73  Distinct  technology  maturity  metrics  for  drugs,  vaccines  and  medical  devices  have  also  been 
established  are  detailed  in  Appendix  H). 
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D.2  SYSTEMS  ENGINEERING  CONTEXT  FOR  IDENTIFYING  CTEs 


CTE  identification  should  be  integral  to  the  systems  engineering  approach  for 
defense  acquisition  programs.  The  following  definition  is  extracted  from  paragraph  4.4. 1 
of  the  Defense  Acquisition  Guidebook'.1 4 


Systems  engineering  is  an  interdisciplinary  approach  encompassing  the 
entire  technical  effort  to  evolve  and  verify  an  integrated  and  total  life  cycle 
balanced  set  of  system,  people,  and  process  solutions  that  satisfy  cus¬ 
tomer  needs.  Systems  engineering  is  the  integrating  mechanism  across 
the  technical  efforts  related  to  the  development,  manufacturing,  verifica¬ 
tion,  deployment,  operations,  support,  disposal  of,  Oand  user  training  for 
systems  and  their  life  cycle  processes.  Systems  engineering  develops 
technical  information  to  support  the  program  management  decision¬ 
making  process.  For  example,  systems  engineers  manage  and  control 
the  definition  and  management  of  the  system  configuration  and  the 
translation  of  the  system  definition  into  work  breakdown  structures. 


Figure  D-l  depicts  one  approach  to  systems  engineering  during  design.  It  portrays 
how  requirements  analysis ,  functional  analysis,  and  design  take  place  iteratively  and 
recursively.  Each  element  influences  and  is  influenced  by  the  others  as  tradeoffs  are  made 
to  discover  the  best  system  solution.  System  operational  requirements,  operational  effec¬ 
tiveness/utility,  and  cost  are  all  considered.  The  functional  analysis  describes  and  evalu¬ 
ates  the  system  in  qualitative  and  quantitative  terms  for  the  functions  that  must  be  done  to 
meet  the  required  performance  characteristics.  Functional  analysis  forms  the  bridge 
between  requirements  and  system  design  where  selections  are  made  among  alternative 
designs  — allocating  scarce  resources  (such  as  cost,  weight,  power,  and  space)  and 
guiding  the  choice  of  optimal  design  points.  As  part  of  this  selection  process,  different 
technologies  are  evaluated  for  maturity,  performance,  cost,  and  manufacturability.  This 
overall  systems  engineering  approach  is  the  sensible  place  to  identify  the  CTEs  and  to 
understand  their  maturity  (i.e.,  their  readiness  for  application  to  the  system  design). 

Two  outcomes  of  the  systems  engineering  approach  are  important  to  CTE  identi¬ 
fication:  (1)  the  functional  architecture,  which  allocates  functional  and  technical  perform¬ 
ance  requirements  and  (2)  the  physical  architecture  (design),  which  shows  the  system 
design  broken  down  into  all  its  constituent  elements  (i.e.,  subsystems  and  components). 
Figure  D-2  displays  the  idea.  The  functional  architecture  establishes  what  the  system 


7  4  Chapter  4  of  the  Defense  Acquisition  Guidebook  provides  a  thorough  coverage  of  systems  engineering. 
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Process  Input 


Figure  D-1.  An  Approach  for  Performing  Front-End  Systems  Engineering 
(Source:  DoD  Systems  Management  College,  Systems  Engineering  Fundamentals, 
Defense  Acquisition  University  (DAU)  Press,  Fort  Belvoir,  VA,  2000) 
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Figure  D-2.  The  Relationship  of  the  Functional  and  Physical  Architectures/Designs 
(Source:  DoD  Systems  Management  College,  Systems  Engineering  Fundamentals, 
Defense  Acquisition  University  (DAU)  Press,  Fort  Belvoir,  VA,  2000) 

accomplishes  in  descriptive  and  quantitative  terms.  It  provides  the  well-defined  frame¬ 
work  around  which  the  physical  architecture  is  conceived  and  designed  and  the  basis 
against  which  the  system  and  its  various  subelements  are  tested.  The  physical  architecture 
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includes  a  representation  of  the  software  and  hardware  “products”  necessary  to  realize  the 
concept.  The  physical  architecture  forms  the  basis  for  design  definition  documentation 
[e.g.,  specifications,  baselines,  and  the  work  breakdown  structure  (WBS)]. 


The  WBS  tool  and  generic  WBSs  applicable  to  seven  specific  categories  of 
defense  materiel  items  are  provided  in  DoD  MIL-HDBK-881,  Work  Breakdown  Struc¬ 
ture ,  dated  2  January  1998.  Figure  D-3  is  a  generic  WBS  for  aircraft.  It  or  any  of  the 
other  six  generic  WBSs  can  be  adapted  to  the  specific  needs  of  many  system  concepts. 
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Figure  D-3.  Generic  Aircraft  WBS 

[Source:  Adapted  from  the  material  ion  Appendix  A  of  MIL-HDBK-881  and  from  DoD 
Systems  Management  College,  Systems  Engineering  Fundamentals,  Defense  Acquisition 
University  (DAU)  Press,  Fort  Belvoir,  VA,  2000  (Figures  8-1  and  9-3] 

The  Defense  Acquisition  Guidebook  specifically  calls  for  use  of  the  WBS  for 
identifying  the  CTEs  for  the  Milestones  B  and  C  TRAs.75  The  WBS  has  several  benefi¬ 
cial  attributes  for  this  purpose: 

•  It  is  readily  available  when  system-engineering  practices  are  used. 

•  It  evolves  with  the  system  concept  and  design. 

•  It  is  composed  of  all  products  that  constitute  a  system  and,  thus,  is  an  apt 
means  to  identify  all  the  technologies  used  by  a  system. 

•  It  relates  to  the  functional  architecture  and,  therefore,  to  the  environment  in 
which  the  system  is  intended  to  be  employed. 

•  It  reflects  the  system  design/architecture  and  the  environment  and  perform¬ 
ance  envelope  for  each  product  in  the  system. 


7^  See  Sections  4. 3. 2.4. 3  and  4. 3. 3. 9.4. 
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While  the  previous  discussion  has  been  for  a  hardware-centric  system,  similar 
approaches  are  present  in  the  systems  engineering  of  Information  Technology  (IT)  sys¬ 
tems,  though  the  terminology  differs.  The  functional  analysis  and  design  synthesis  por¬ 
trayed  in  Figure  D-l  are  encompassed  in  the  IT  architectural  design  process. 

The  DoD  Architecture  Framework  (DoDAF)76  defines  a  common  approach  for 
DoD  architecture  description,  development,  presentation,  and  integration.  It  describes 
three  related  views  of  architecture: 

1.  The  Operational  View  (OV)  identifies  what  needs  to  be  accomplished  and 
who  does  it. 

2.  The  Systems  View  (SV)  relates  systems  and  characteristics  to  operational 
needs. 

3.  The  Technical  Standards  View  (TV)  prescribes  standards  and  conventions. 

Products  within  this  framework  can  be  associated  with  the  systems  engineering  func¬ 
tional  and  physical  architectures  described  in  this  section. 

According  to  Buede,  a  systems  engineering  functional  architecture  “contains  a 
hierarchical  model  of  the  functions  performed  by  the  system,  the  system’s  components, 
and  the  system’s  configuration  items  (CIs);  the  flow  of  informational  and  physical  items 
from  outside  the  system  through  the  transformational  process  of  the  system’s  functions 
and  on  the  waiting  external  systems  being  serviced  by  the  system;  a  data  model  of  the 
system’s  items;  and  a  tracing  of  input/output  requirements  to  both  the  system’s  functions 
and  items.”  77  IT  systems  engineering  creates  a  data  model  that  exposes  data  types  and 
their  relationships.  This  data  model  includes  a  description  of  data  flow  (i.e.,  how  the 
activities  of  the  IT  system  affect  the  data)  and  the  distribution  of  computational  processes 
over  the  system.  This  is  analogous  to  the  functional  architecture  for  a  hardware-centric 
system.  There  are  seven  associated  DoDAF  products: 

1.  OV-2,  Operational  Node  Connectivity  Description 

2.  OV-3,  Operational  Information  Exchange  Matrix 

3.  OV-5,  Operational  Activity  Model 

4.  OV-6,  Operational  Activity  Sequence  and  Timing  Description 


76  DoD  Architectural  Framework,  Version  1.0,  “Volume  I:  Definitions  and  Guidelines,”  “Volume  II: 
Product  Descriptions,”  9  February  2004. 

77  Dennis  M.  Buede,  The  Engineering  Design  of  Systems:  Models  and  Methods,  John  Wiley  &  Sons,  Inc., 
2000,  p.  175. 
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5.  OV-7,  Logical  Data  Model 

6.  SV-4,  Systems  Functionality  Description 

7.  SV-5,  Operational  Activity  to  System  Functionality  Traceability  Matrix. 

Buede  describes  the  systems  engineering  physical  architecture  as  “a  hierarchical 
description  of  the  resources  that  comprise  the  system.  . . .  [It]  provides  resources  for  every 
function  identified  in  the  functional  architecture.”  78  The  IT  analog  is  captured  in  several 
DoDAF  products  in  the  SV: 

•  SV-1,  Systems  Interface  Description 

•  SV-2,  Systems  Communications  Description 

•  SV-3,  Systems-Systems  Matrix 

•  SV-7,  System  Performance  Parameters  Matrix. 

This  hardware  technology  is  typically  described  as  a  systems  architecture  and  often  exists 
at  several  levels  of  detail,  ranging  from  a  prototype  architecture  through  a  detailed 
architecture. 

D.3  PROCEDURES  AND  PRACTICES  FOR  IDENTIFYING  CTEs 

D.3.1  Overall  Description 

The  management  process/ 
procedure  for  CTE  identification  is  as 
important  as  the  technical  task 
because  it  adds  to  the  credibility  of 
the  resulting  CTE  list.  While  the  Pro¬ 
gram  Manager  (PM)  holds  the  basic  responsibility  for  identifying  the  CTEs,  ultimately, 
the  Component  Acquisition  Executive  (CAE)  endorses  this  list  to  the  Office  of  the  Sec¬ 
retary  of  Defense  (OSD)  as  part  of  the  information  forwarded  before  the  Milestone  B  and 
Milestone  C  reviews. 

From  a  management  process/procedure  perspective,  CTE  identification  should  be 
a  two-step  process.  In  the  first  step,  the  CTE  definition  is  applied  across  the  system’s 
WBS  or  architecture  to  identify  critical  technology  candidates.  This  process  should  be 


Best  Practice 

Use  the  WBS  or  system  architecture 
to  identify  CTEs. 


78  Dennis  M.  Buede,  The  Engineering  Design  of  Systems:  Models  and  Methods,  John  Wiley  &  Sons,  Inc., 
2000,  pp.  215-216. 
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thorough,  disciplined,  and  conservative.  Any  questionable  technology  should  be  identi¬ 
fied  as  a  candidate  CTE.  For  these  questionable  technologies,  the  information  required  to 
resolve  their  status  should  be  documented.  The  PM,  the  government  program  office  staff, 
and  the  system  contractors— the  people  best  informed  about  the  system— should  lead  the 
first  step.  The  second  step  consists  of  resolving,  where  possible,  the  status  of  technologies 
in  question  by  filling  the  information  gaps  noted  in  the  first  step.  An  independent  panel  of 
experts  convened  by  the  Component  Science  and  Technology  (S&T)  Executive  should 
conduct  the  second  step. 

All  individuals  involved  in  these  steps  should  be  familiar  with 

•  CTE  identification  in  the  context  of  a  TRA  and  its  importance  to  the  techni¬ 
cal  and  programmatic  success  of  the  program 

•  The  concept  of  the  WBS  or  systems  architecture  as  a  complete  description  of 
the  products/things  that  comprise  a  system 

•  The  distinction  between  hardware,  software,  and  manufacturing  technologies 
and  the  metrics  that  evaluate  their  maturity  (as  described  in  Appendix  C) 

•  The  affordability  and  production  criteria  for  CTEs 

•  The  role  that  “environment”  has  in  identifying  CTEs. 

The  technical  task  involves  the  use  of  a  series  of  questions  to  test  whether  the 
CTE  definition  applies.  For  a  technology  to  be  critical,  the  answer  to  one  of  the  following 
questions  must  be  “yes”: 

•  Does  the  technology  directly  impact  an  operational  requirement? 

•  Does  the  technology  have  a  significant  impact  on  an  improved  delivery 
schedule? 

•  Does  the  technology  have  a  significant  effect  on  the  system’s  affordability? 

•  If  this  is  a  spiral  development,  is  the  technology  essential  to  meet  the  spiral 
deliverables? 

In  addition,  the  answer  to  one  of  the  following  questions  must  also  be  “yes”: 

•  Is  the  technology  new  or  novel? 

•  Is  the  technology  modified? 

•  Has  the  technology  been  repackaged  so  that  a  new  relevant  environment  is 
realized? 

•  Is  the  technology  expected  to  operate  in  an  environment  and/or  achieve  a  per¬ 
formance  beyond  its  original  design  intention  or  demonstrated  capability? 
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The  environment  in  which  the  system  will  operate  plays  a  significant  role  in 
answering  these  last  four  questions.  Subsection  D.3.2  provides  a  more  detailed  explana¬ 
tion  of  that  role. 

D.3.2  Environments 

Consideration  of  the  environ¬ 
ment  is  important  for  CTE  identifica¬ 
tion.  For  a  CTE  to  be  assessed  at  TRL  6, 
the  required  level  at  Milestone  B,  it 
must  have  been  demonstrated  in  a  rele¬ 
vant  environment.  For  a  CTE  to  be 
assessed  at  TRL  7,  the  required  level  at 
Milestone  C,  it  must  have  been  demon¬ 
strated  in  an  operational  environment.  79 

Generally,  the  requirement  statement  for  the  system  will  provide  some  description 
of  the  environment  in  which  the  system  is  expected/required  to  operate.  This  can  be 
called  the  external  or  imposed  environment.  It  may  be  natural  or  man-made,  friendly  or 
hostile  (e.g.,  weather,  terrain,  friendly  and  hostile  jamming,  enemy  fire,  and  so  forth). 
Another  environment— the  one  generally  more  important  for  identifying  and  evaluating 
CTEs— can  be  called  the  internal  or  realized  environment.  It  is  derived  from  the  perform¬ 
ance  required  of  each  design  item  (product,  subsystem,  component,  WBS  element).  The 
design  analysis  should  include  the  required  or  expected  performance  envelope  and  con¬ 
ditions  for  each  WBS  element. 

Categories  of  environment  and  their  identification  are  discussed  below  briefly. 
The  intent  is  to  provide  some  ideas  for  factoring  environments  into  CTE  identification. 

Environments  will  likely  include 

•  Physical  Environment.  For  instance,  mechanical  components,  processors, 
servers,  and  electronics;  kinetic  and  kinematic;  thermal  and  heat  transfer; 
electrical  and  electromagnetic;  climatic— weather,  temperature,  particulate; 
network  infrastructure 

•  Logical  Environment.  For  instance,  software  (algorithm)  interfaces;  secu¬ 
rity  interfaces;  Web-enablement 


Best  Practice 

Information  for  CTE  identification 
should  include  results  of  design 
analyses  that  define  performance 
expectations  of  components  and 
the  data  and  physical  conditions  in 
which  they  operate. 


79 


Section  3  and  Appendix  C  of  this  Deskbook  present  a  more  detailed  discussion  of  TRLs. 


D-ll 


•  Data  Environment.  For  instance,  data  formats  and  databases;  anticipated 
data  rates,  data  delay  and  data  throughput;  data  packaging  and  framing 

•  Security  Environment.  For  instance,  connection  to  firewalls;  security  appli¬ 
ques;  rates  and  methods  of  attack 

•  User  and  Use  Environment.  For  instance,  scalability;  upgradability;  user 
behavior  adjustments;  user  interfaces;  organizational  change/realignments 
with  system  impacts;  implementation  plan. 

Various  environments 
not  listed  previously  are  almost 
certain  to  be  relevant  to  any 
specific  system.  If  the  SV  and 
OV  of  the  design/architecture 
have  been  used  to  identify 
potential  CTEs,  they  can  also 
be  used  to  help  identify  the 
environment,  especially  the 
Logical  and  Data  Environ¬ 
ments.  Requirements  can  also 
be  used  to  help  identify  the 
environment.  In  addition,  inter¬ 
operability  documents  and  Interface  Control  Documents  (ICDs)  should  be  used  to  iden¬ 
tify  the  environments  in  which  the  candidate  CTEs  will  operate.  Key  questions  that  can 
help  guide  the  definition  of  the  environment  for  the  CTE  candidates  might  include  the 
following: 

•  Is  the  physical/logical/data  environment  in  which  this  CTE  has  been  demon¬ 
strated  similar  to  the  intended  environment?  If  not,  how  is  it  different?  Is  the 
difference  important? 

•  Is  the  CTE  going  to  be  operating  at  or  outside  of  the  usual  performance  enve¬ 
lope?  Do  the  design  specifications  address  the  behavior  of  the  CTE  under 
these  conditions?  What  is  unique  or  different  about  this  proposed  operations 
environment? 

•  Do  test  data,  reports,  or  analyses  that  compare  the  demonstrated  environment 
to  the  intended  environment  exist?  If  modeling  and  simulation  (M&S)  is  an 
important  aspect  of  that  comparison,  are  the  analysis  techniques  common  and 
generally  accepted? 


Best  Practice 

People  with  the  requisite  technical  knowl¬ 
edge  and  the  independence  needed  to 
make  a  good  judgment  should  guide  the 
actual  set  of  questions  asked  for  each 
CTE  candidate.  The  PM  and  the  suppliers 
should  present  clear,  convincing,  and 
succinctly  summarized  data  that  show 
what  is  known/not  known  about  the  envi¬ 
ronment  and  should  explain  the  similari¬ 
ties  and  dissimilarities  between  the 
expected/demonstrated  environments . 
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The  following  subsections  give  more  examples  of  the  kinds  of  questions  and 
sources  of  information  that  can  be  used  to  help  define  the  environment. 

D.3.2.1  Defining  the  Physical  Environment 

Representative  questions  that  will  be  helpful  in  identifying  the  physical  environ¬ 
ment  (and  whether  it  is  new  or  novel)  for  the  candidate  CTE  include  the  following: 

•  What  are  the  expected  conditions  (vibration,  movement,  exposure  to  heat, 
and  so  forth)  in  which  the  candidate  CTE  will  reside?  Do  any  data  or  analysis 
show  how  the  demonstrated  environment  resembles  the  expected  extremes? 

•  What  is  the  electromagnetic  environment  in  which  the  candidate  CTE  will 
reside?  Has  it  been  tested  or  demonstrated  in  that  full  environment? 

•  What  is  the  server/processor/network  environment?  How  does  the  designer 
know  that  the  CTE  will  operate  in  that  environment? 

•  What  interfaces  will  be  used?  How  do  they  compare  with  interfaces  used 
previously? 

•  What  network  infrastructure  will  be  used?  How  will  the  load  over  this  infra¬ 
structure  be  affected  by  the  new  system? 

D.3.2.2  Defining  the  Logical  and  Data  Environments 

Operational  and  systems  architectures  can  be  used  to  help  determine  the  Logical 
and  Data  Environments  in  which  the  CTE  will  operate.  Designs  or  WBSs  can  also  be 
useful.  Whether  the  CTE  is  a  commercial  off-the-shelf/government  off-the-shelf  (COTS/ 
GOTS)  software  package  or  is  a  network  card,  the  CTE  has  a  logical  relationship  to  other 
systems  and  to  the  outside  world.  Those  logical  relationships— the  Logical  Environ¬ 
ment-may  or  may  not  be  similar  to  the  proposed  DoD  environment.  Furthermore,  the 
databases  and  their  configuration  (e.g.,  partitioned,  replicated,  standalone)  and  the  antici¬ 
pated  transaction  rates  in  the  proposed  DoD  system  may  be  different  from  previous 
environments  in  which  the  CTE  has  been  used.  These  differences  should  be  documented 
and  evaluated  for  relevance.  Sometimes,  a  developer  may  use  an  interface  simulation  or 
ersatz  data  to  try  to  replicate  the  logical  and  data  environments. 

Relevant  questions  that  may  be  helpful  in  identifying  and  evaluating  the  logical 
and  data  environments  for  the  candidate  CTE  include  the  following: 

•  What  are  the  expected  logical  relationships  between  the  CTE  and  the  rest  of 
the  system?  The  outside  world? 

•  What  are  the  expected  data  rates?  the  expected  data  formats? 
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D.3.2.3  Defining  the  Security  Environment 

Frequently,  the  security  environment  will  differ  from  the  environment  in  which  a 
CTE  has  been  demonstrated,  especially  in  COTS  systems.  Thus,  every  CTE  candidate 
system  should  include  a  careful  definition  of  the  security  environment  in  which  it  will 
reside. 

The  security  environment  includes  hardware  components  (e.g.,  firewalls,  network 
gateways),  logical  components,  (e.g.,  potential  virtual  circuits),  and  data.  Requirements 
for  the  security  environment  can  often  be  derived  from  IA  requirements.  In  addition,  the 
systems  architecture  can  be  a  source  of  information. 

The  rates  and  methods  of  attack  during  wartime  and  peacetime  may  also  be  ele¬ 
ments  of  the  security  environment.  Technical  experts  in  IT  and  network  security  can  be 
helpful  in  defining  and  evaluating  the  security  environment.  An  important  question  is  the 
anticipated  differences  in  environment  in  wartime  as  compared  with  the  environments  in 
peacetime.  Often,  the  security  requirements  tighten  during  wartime,  and  evaluators 
should  take  care  in  defining  those  differences. 

D.3.2.4  Defining  the  User  and  Use  Environment 

The  user  and  use  environments  are  closely  tied  to  the  physical  environments. 
They  deal  with  the  interactions  between  the  human  users  and  the  physical  system  over  a 
collection  of  many  possible  scenarios  and  sequences.  Relevant  questions  for  better  under¬ 
standing  the  user  and  use  environment  for  identifying  CTEs  include  the  following: 

•  What  is  the  expected  user  environment?  How  do  the  number  of  users  and  the 
way  in  which  they  will  use  the  system  compare  with  what  has  been  done 
before? 

•  What  are  the  expectations  for  growth  over  time?  Is  it  likely  that  usage  will 
increase  significantly  beyond  those  expectations? 

•  What  organizational  changes  are  anticipated?  What  are  the  foreseeable  sys¬ 
tem  impacts  based  on  a  new  organizational  structure? 

•  How  will  users’  jobs  be  affected?  Will  the  changes  be  gradual  or  abrupt? 
What  is  the  expected  user  reaction? 

•  How  much  resistance  to  change  is  anticipated?  Are  plans  in  place  to  mitigate 
such  resistance? 

•  Has  the  learning  curve  for  adapting  to  the  new  system  been  anticipated  and 
have  preparations  been  made  to  address  this  issue?  Is  training  in  place? 
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Have  all  interfaces  between  existing  processes  and  the  new  system  changed 
correspondingly? 

Has  an  implementation  or  roll-out  plan  been  considered  for  the  new  system? 


D.4  REPRESENTATIVE  QUESTIONS  FOR  IDENTIFYING  CTEs 

Identifying  CTEs  depends  on  effective  questioning.  While  a  universal  list  of 
“right”  questions  does  not  exist,  the  following  discussion  provides  typical  questions  for 
several  categories  of  systems  and  suggests  the  nature  of  what  is  intended.  Every  actual 
system  should  use  a  relevant  set  of  questions  tailored  to  its  application. 

D.4.1  Aircraft 

A  few  of  the  pertinent  questions  to  ask  when  trying  to  identify  the  CTEs  for 
aircraft  development  are  as  follows:. 

•  Aerodynamic  configuration.  Does  the  design  incorporate  a  configuration 
that  has  not  been  used  in  flight?  How  similar  is  the  configuration  to  that  of 
aircraft  that  are  successful?  Does  the  configuration  impose  limitations  on 
control  authority,  stability,  structural  rigidity,  or  strength?  Is  stability  accept¬ 
able  at  high  angles  of  attack?  Are  stability  and  control  acceptable  during 
configuration  changes  in  flight? 

•  Flight  performance.  Is  the  lift-to-drag  (L/D)  ratio  being  used  in  range  calcu¬ 
lations  consistent  with  that  being  achieved  by  operating  aircraft?  Has  this 
L/D  ratio  been  confirmed  by  wind  tunnel  tests  corrected  to  full-scale, 
trimmed  conditions?  Are  takeoff  and  landing  distances  based  on  achievable 
lift  coefficients  and  installed  thrust? 

•  Airframe  structure  and  weight.  Is  the  structural  weight  fraction  consistent 
with  operating  aircraft  of  the  same  type?  Are  lower  fractions  justified  by  use 
of  more  efficient  materials  or  structural  designs?  Do  the  materials  and  struc¬ 
tures  have  stiffness  and  fatigue  properties  suitable  to  the  application?  Has 
this  been  demonstrated  with  full-scale  sections  and  representative  loads? 

•  Propulsion.  Do  the  engine  hot  sections  rely  on  new  materials?  Have  these 
been  tested  to  the  temperatures,  loads,  and  dynamic  environment  of  expected 
flight?  Are  the  results  for  thrust  and  specific  fuel  consumption  (SFC)  from 
ground  tests  consistent  with  the  estimates?  Have  the  inlets  been  tested  at 
flight  flow  rates? 

•  Rotors  and  hubs.  Has  the  rotor  type  been  used  before  in  a  similar  applica¬ 
tion?  Has  testing  been  limited  to  static  conditions?  Has  a  similar  type  of  rotor 
been  tested  at  a  relevant  scale?  Is  there  a  test  basis  for  the  durability  estimates 
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for  the  rotor  and  hub?  Do  the  cyclic  and  collective  control  mechanisms  differ 
from  common  practice?  How  have  they  been  tested? 

•  Mission  equipment.  The  appropriate  questions  differ  greatly  for  the  different 
roles  aircraft  play.  Advanced  technology  might  be  incorporated  in  weapon 
carriage  and  employment,  in  cargo  handling,  in  surveillance,  in  communica¬ 
tions,  and  elsewhere.  General  questions  include  the  following:  What  limits 
the  operational  effectiveness  of  this  design?  How  is  advanced  technology 
contributing  to  more  effective  performance  of  the  aircraft  mission?  Are  any 
of  these  technologies  unproven  in  this  application? 

•  Manufacturing  technology.  The  identification  of  manufacturing  technology 
CTEs  will  require  an  analysis  to  determine  the  availability  of  essential  raw 
materials,  special  alloys,  composite  materials,  components,  tooling,  and  pro¬ 
duction  test  equipment  required  for  (1)  the  sustained  production  of  a  system 
fully  capable  of  meeting  performance  objectives  established  for  the  system, 
(2)  the  uninterrupted  maintenance  and  repair  of  the  system,  and  (3)  the  sus¬ 
tained  operation  of  the  system.  Pertinent  questions  include  the  following: 
Will  the  technology  require  the  use  of  advanced  manufacturing  technology, 
processes,  and  systems  during  the  research  and  development  (R&D)  and  the 
production  phases  of  the  program?  Has  the  technology  been  characterized  in 
a  manufacturing  environment?  Has  the  manufacturing  technology  been  dem¬ 
onstrated  on  a  similar  system?  Will  the  manufacturing  technology  require  a 
scale-up  effort  for  the  proposed  system  being  developed  and  produced? 

D.4.2  Ground  Vehicles 

Some  suggestions  are  provided  to  indicate  ways  to  undertake  the  task  of  iden¬ 
tifying  CTEs  for  ground  vehicles.  Usually  (but  not  necessarily)  the  vehicle  system  under 
consideration  is  similar  to  an  existing  class  of  vehicles  and  their  functions.  Military  sys¬ 
tems  are  usually  categorized  as  combat  vehicles  (e.g.,  tanks),  tactical  vehicles  [e.g.,  High 
Mobility  Multipurpose  Wheeled  Vehicles  (HMMWVs)],  or  utility  vehicles  (e.g.,  sedans 
or  special-purpose  vehicles).  A  first  step  for  CTE  identification  is  to  exploit  the  associa¬ 
tion  and  the  functional  similarities  that  exist  between  existing  systems  and  the  proposed 
system  by  characterizing  (quantitatively  wherever  possible)  the  functions  of  the  new 
system  and  those  of  comparative  existing  systems.  The  second  step  is  to  carry  out  com¬ 
parisons  of  the  proposed  technologies  of  the  new  system  to  identify  whether  these  tech¬ 
nologies  are  new  or  just  new  or  novel  in  application.  Of  course,  the  possibility  exists  that 
this  comparison  process  does  not  cover  all  new  technologies.  In  those  instances,  the  tech¬ 
nologies  not  covered  will  require  alternative  ways  to  assess  whether  they  are  critical.  The 


D-16 


fact  that  they  have  not  been  used  previously  is  a  good  indicator  that  they  are  candidate 
CTEs. 


As  an  example,  a  few  useful  questions  for  a  new  fighting  vehicle  system  are 
listed.  These  questions  address  the  principal  functions  of  mobility,  firepower,  and  protec¬ 
tion.  In  an  actual  case,  a  set  of  questions  could/should  be  developed  around  a  WBS  built 
upon  the  template  for  vehicles  found  in  MIL  HDBK-881.  Of  course,  special  mission 
equipment  and  other  items  should  also  be  considered. 

•  Mobility  (e.g.,  WBS  elements:  power  package/drive  train,  suspension/ 
steering).  How  do  mobility  characteristics  (range,  speed,  agility,  endurance 
and  so  forth)  compare  with  existing  vehicles?  Is  the  suspension  system 
proven  for  the  weight  and  mobility  required  of  the  concept  system?  Has  the 
suspension  system  been  proven  to  provide  a  robust,  reliable,  and  stable  plat¬ 
form  for  stationary  and  on-the-move  firing  for  the  type  of  armaments  systems 
intended  for  the  concept  vehicle?  Are  the  engine  characteristics  (power  per 
unit  weight,  SFC,  cooling  and  thermal  signature  characteristics,  and  so  forth) 
proven  in  service?  Are  the  power  train  elements  new  or  in  new  environments 
or  with  extended  performance  envelopes? 

•  Firepower  (e.g.,  WBS  elements:  armament,  fire  control,  automatic 
loading).  Are  the  weapons  new?  Is  new  ammunition  to  be  developed?  Are 
the  natures  of  ammunition  to  be  developed  new?  Will  there  be  an  autoloader? 
If  so,  is  it  new?  Has  ammunition  and  autoloader  compatibility  been  estab¬ 
lished?  Has  a  weapon  that  has  the  intended  characteristics  ever  been  mated 
with  a  platform  of  the  weight  and  structure  characteristics  of  the  vehicle 
platform?  Are  firing  data  available  on  force  and  motion  characteristics  of  the 
weapon  for  all  the  intended  natures  of  ammunition? 

•  Protection  (e.g.,  WBS  elements:  hull/frame,  turret  assembly).  Are  full- 
scale  data  available  to  demonstrate  that  the  intended  passive  protection  is 
adequate  for  all  features  and  required  aspects  of  the  design  configuration?  If 
not,  what  are  the  alternative  approaches  and  what  data  are  available  to  dem¬ 
onstrate  that  they  meet  the  need?  Are  reactive  armor  applications  intended 
and  are  data  available  to  allow  a  flexible  design  that  meets  system  needs? 
Does  the  reactive  armor  meet  logistic  requirements  (e.g.,  are  there  insensitive 
explosive  mandates)?  Is  the  use  of  an  active  protection  system  intended?  If 
so,  what  data  are  available  to  demonstrate  its  efficacy? 

•  Manufacturing  technology.  The  identification  of  manufacturing  technology 
CTEs  will  require  an  analysis  to  determine  the  availability  of  essential  raw 
materials,  special  alloys,  composite  materials,  components,  tooling,  and  pro¬ 
duction  test  equipment  required  for  (1)  the  sustained  production  of  a  system 
fully  capable  of  meeting  performance  objectives  established  for  the  system, 
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(2)  the  uninterrupted  maintenance  and  repair  of  the  system,  and  (3)  the  sus¬ 
tained  operation  of  the  system.  Pertinent  questions  include  the  following: 
Will  the  technology  require  the  use  of  advanced  manufacturing  technology, 
processes,  and  systems  during  the  R&D  and  the  production  phases  of  the 
program?  Has  the  technology  been  characterized  in  a  manufacturing  environ¬ 
ment?  Has  the  manufacturing  technology  been  demonstrated  on  a  similar 
system?  Will  the  manufacturing  technology  require  a  scale-up  effort  for  the 
proposed  system  being  developed  and  produced? 

D.4.3  Missiles 

To  discover  CTEs  in  a  missile  development,  the  following  questions  might  be 
helpful: 


•  Guidance  and  control.  Has  the  type  of  guidance  under  consideration  been 
used  before?  If  so,  was  it  successful  in  the  similar  application?  Does  the  field 
of  view  (FOV),  field  of  regard  (FOR),  scan  rate,  slew  rate,  sensitivity,  acuity, 
or  any  other  performance  parameters  exceed  what  has  been  achieved  in 
affordable  guidance  systems?  Has  the  guidance  system  been  tested  in  proto¬ 
type  form?  Has  it  been  tested  from  a  tower,  in  captive  carry,  or  in  flight?  Has 
it  been  tested  against  realistic  targets  in  realistic  environments?  Are  the  sen¬ 
sor  range  and  the  missile  control  time  constant  compatible  with  the  dynamics 
of  the  end  game?  How  has  this  been  established? 

•  Propulsion  and  structure.  Is  there  a  propellant  that  can  meet  the  specific 
impulse  requirement  and  have  acceptable  safety  characteristics,  burn  rates, 
physical  characteristics,  and  cost?  What  size  batches  of  this  propellant  have 
been  made?  What  size  test  motors  have  been  fired?  Has  the  combination  of 
case,  insulation,  grain  support,  and  grain  configuration  ever  been  used  in  a 
rocket  motor?  Does  the  design  have  any  special  features  (e.g.,  multiple  burn, 
throttling,  air-burning,  case-consuming,  throatlessness)? 

•  Manufacturing  technology.  The  identification  of  manufacturing  technology 
CTEs  will  require  an  analysis  to  determine  the  availability  of  essential  raw 
materials,  special  alloys,  composite  materials,  components,  tooling,  and 
production  test  equipment  required  for  (1)  the  sustained  production  of  a 
system  fully  capable  of  meeting  performance  objectives  established  for  the 
system,  (2)  the  uninterrupted  maintenance  and  repair  of  the  system,  and 
(3)  the  sustained  operation  of  the  system.  Pertinent  questions  include  the  fol¬ 
lowing:  Will  the  technology  require  the  use  of  advanced  manufacturing 
technology,  processes,  and  systems  during  the  R&D  and  the  production 
phases  of  the  program?  Has  the  technology  been  characterized  in  a  manufac¬ 
turing  environment?  Has  the  manufacturing  technology  been  demonstrated 
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on  a  similar  system?  Will  the  manufacturing  technology  require  a  scale-up 
effort  for  the  proposed  system  being  developed  and  produced? 

D.4.4  Ships,  Submarines,  and  Naval  Weapons  Systems 

The  at-sea  environment  poses  unique  challenges  to  new  technologies  and  systems. 
The  new  system  will  have  some  questions  that  apply  to  all  combat  systems  and  other 
questions  that  are  appropriate  for  all  hull,  mechanical,  and  electrical  systems.  There  will 
also  be  unique  questions  for  all  surface  ship  systems  and  others  that  apply  to  all  subma¬ 
rine  systems. 

•  Combat  systems.  Has  the  weapon  system  been  tested  at  sea  to  establish  its 
firing  accuracy  in  a  realistic  environment?  Has  the  affect  of  ship  motion  and 
weather  variables  on  targeting  been  taken  into  account?  Has  the  weapon  been 
cleared  to  be  placed  on  board  a  ship  or  submarine  by  the  Weapon  Systems 
Explosive  Safety  Review  Board  (WSERB)?  Does  the  weapon  warhead  meet 
insensitive  munitions  requirements?  Has  the  sensor  system  been  tested  in 
realistic  at-sea  conditions  for  wave  motions  and  accelerations?  Are  batteries 
and  power  supplies  needed  by  the  sensor  system  compatible  with  the  ship’s 
power  grid?  Is  the  system  safe  or  does  it  present  hazards  in  case  of  fire  or 
shock?80  Has  the  weapon  or  sensor  system  been  evaluated  for  maintenance 
requirements  and  logistics  needs  since  the  ship  is  a  closed  system  that  must 
carry  its  own  spares? 

•  Ship  and  submarine  hull,  mechanical,  and  electrical  systems.  Does  the 
new  system  or  hull  itself  use  new  materials?  Have  these  materials  been 
evaluated  for  corrosion  at  sea?  How  does  the  weight  of  a  new  hull  compare 
with  previous  designs?81  If  the  new  hull  system  comes  from  a  commercial 
application,  has  it  been  evaluated  for  military  usage?  In  the  case  of  a  subsys¬ 
tem,  has  it  been  to  sea  on  a  ship  or  submarine  previously?  For  a  new  hull  or  a 
new  material,  can  it  withstand  the  effect  of  a  collision  or  grounding  incident? 
Does  the  new  system  add  to  the  vulnerability  of  the  ship  to  withstand  damage 
without  sinking?82  For  new  propulsion  systems,  does  the  new  system  provide 
an  improvement  in  propulsive  efficiency?  Does  it  increase  or  decrease  the 
ship  or  submarine  signature?  Does  the  new  system  increase  the  draft  of  the 
ship,  thus  limiting  the  ports  in  which  it  can  operate?  Does  the  propulsion 
system  cavitate  during  operation,  thus  reducing  efficiency? 


80  Some  batteries  are  not  allowed  on  submarines  because  of  their  reaction  to  fire. 

81  The  structural  weight  fraction  should  be  within  historical  bounds. 

82  Strict  rules  apply  to  new  hulls  and  major  subsystems. 
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•  Submarine-specific  issues.  Has  the  new  system  been  tested  at  depth?  Does  it 
meet  the  Submarine  Safety  Certification  Program  (SUBSAFE)  require¬ 
ments?83  Does  the  new  system  add  to  the  submarine  acoustic  or  nonacoustic 
signature  in  any  way?  Does  the  system  generate  underwater  sound  that  is 
detrimental  to  marine  life? 

•  Surface-ship-specific  issues.  Will  the  system  or  subsystem  stand  up  to  the 
motions  and  accelerations  caused  by  waves?  Will  the  system  or  subsystem 
increase  the  ship’s  drag  in  any  way?  Will  the  system  or  subsystem  have  an 
environmentally  unacceptable  discharge? 

•  Manufacturing  technology.  The  identification  of  manufacturing  technology 
CTEs  will  require  an  analysis  to  determine  the  availability  of  essential  raw 
materials,  special  alloys,  composite  materials,  components,  tooling,  and  pro¬ 
duction  test  equipment  required  for  (1)  the  sustained  production  of  a  system 
fully  capable  of  meeting  performance  objectives  established  for  the  system, 
(2)  the  uninterrupted  maintenance  and  repair  of  the  system,  and  (3)  the  sus¬ 
tained  operation  of  the  system.  Pertinent  questions  include  the  following: 
Will  the  technology  require  the  use  of  advanced  manufacturing  technology, 
processes,  and  systems  during  the  R&D  and  the  production  phases  of  the 
program?  Has  the  technology  been  characterized  in  a  manufacturing  environ¬ 
ment?  Has  the  manufacturing  technology  been  demonstrated  on  a  similar 
system?  Will  the  manufacturing  technology  require  a  scale-up  effort  for  the 
proposed  system  being  developed  and  produced? 

D.4.5  Information  Systems 

•  General  questions  (particularly  for  COTS).  Does  this  CTE  claim  to  imple¬ 
ment  standards  that  provide  critical  functionality?  How  was  the  compliance 
to  these  standards  verified?  Is  there  familiarity  with  the  element  from  other 
projects?  What  aspects  of  the  system  design  are  dependent  on  unique  features 
or  particular  versions  of  the  CTE?  Will  these  unique  features  be  sustained  in 
future  versions  of  the  CTE?  Will  this  CTE  be  modified,  tailored,  extended,  or 
enhanced  from  its  original  state?  Who  will  perform  these  modifications? 
How  complex  are  these  modifications?  Does  this  CTE  depend  on  other  sys¬ 
tems?  Does  the  CTE  conform  with  size,  weight,  and  power  requirements? 

•  Terminal  hardware.  Terminal  hardware  consists  of  video  displays,  audio/ 
sound  systems,  keyboards,  touch-screen  terminals,  personal  digital  assistants 
(PDAs)  and  so  forth.  Are  there  extenuating  physical  environment  considera¬ 
tions  for  size,  weight,  visibility  in  daylight,  usability? 


83  SUBSAFE  is  a  specific  and  rigorous  testing  procedure. 
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•  Processing  hardware.  Processing  hardware  consists  of  processors,  memory, 
servers,  supercomputers,  mainframes,  blade  servers,  and  so  forth.  Are  needed 
software  development  environments  supported?  Are  there  any  significant 
changes  to  the  operating  system  and  other  systems  software?  Are  processors 
able  to  handle  average  and  peak  processing  loads? 

•  Storage  hardware.  Storage  hardware  consists  of  disk  drives,  magnetic  tapes, 
redundant  array  of  inexpensive  disks  (RAID),  controllers,  and  so  forth.  How 
is  storage  being  connected  to  the  processing  hardware?  Is  storage  balanced 
with  processing  capacity?  How  will  storage  scale  with  increasing  processing 
capacity? 

•  Networking  hardware.  Networking  hardware  consists  of  routers,  switches, 
access  points,  network  interface  cards  (NICs),  local  area  network/wide  area 
network  (LAN/WAN)  components,  storage  area  network  (SAN)  components, 
and  so  forth.  Do  requirements  for  bandwidth,  delay,  loss,  and  availability 
imply  that  new  or  modified  hardware  is  required?  Is  wireless  performance 
acceptable  in  the  expected  electromagnetic  environment?  Is  the  network  able 
to  grow  in  physical  size  and  bandwidth  while  still  satisfying  key  performance 
requirements? 

D.4.6  Networked  Communications  Systems 

The  following  questions  can  help  identify  CTEs  in  a  networked  communications 

system: 

•  Do  the  requirements  for  throughput,  data  latency,  security,  or  reliability 
imply  that  a  new  or  novel  technology  is  required?  Have  the  network  routers 
been  used  before  within  the  required  performance  envelope?  Are  new  or 
novel  media  access  control,  coding,  or  routing  algorithms  needed?  Is  the 
multiplexing  schema  new?  Is  the  topology  (logical  and  hardware)  new?  Do 
the  peak  and  average  data  rates  require  new  hardware  or  algorithms  in  the 
system? 

•  If  the  network  includes  wireless  technology,  have  the  wireless  devices  been 
used  in  the  anticipated  electromagnetic  environment?  Does  the  way  in  which 
data  sources  or  uses  interface  to  the  network  imply  a  need  for  a  new  interface 
(logical  or  hardware)?  Does  the  ICD  identify  any  interfaces  that  are  new  or 
novel? 

•  If  the  network  includes  commercially  available  elements,  such  as  Asynchro¬ 
nous  Transfer  Mode  (ATM)84  and  optical  components,  or  Ethernet,  and  so 


84  Broadband  switching  and  transmission  technology. 
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forth,  have  these  elements  been  demonstrated  for  their  intended  use?  That  is, 
do  they  support  the  data  rates,  switching  schema,  routing,  and  so  forth?  Do 
the  IA  requirements  create  a  new  or  novel  security  environment? 

•  Do  requirements  for  scalability  and  the  capability  to  upgrade  imply  the  need 
for  new  algorithms?  Does  the  scale  of  the  system  imply  a  new  environment 
for  the  network? 

D.4.7  Business  Systems 

DoD  business  systems  often  use  COTS  products  assembled  together  to  achieve  a 
new  capability.  Some  relevant  questions  are  as  follows: 

•  Are  the  logical  and  data  environments  for  each  COTS  element  new  or  novel? 
Are  there  special  data  synchronization  requirements  or  needs  that  imply  the 
need  for  new  wrapper  algorithms?  Has  the  COTS  system  run  in  the  operating 
system  environment  or  on  the  target  workstations  and  servers? 

•  Is  a  new  suite  of  hardware  (servers,  networks,  and  so  forth)  needed  to  run  the 
business  system?  Will  the  interfaces  for  the  server  require  a  new  or  novel 
hardware  or  software  technology?  Will  new  processors  be  required?  If  so, 
will  these  processors  support  the  anticipated  speeds? 

•  Does  the  IA  requirement  imply  a  new  security  environment?  Have  the 
selected  COTS  products  been  demonstrated  or  tested  with  the  IA  technolo¬ 
gies  chosen  for  the  system?  Do  the  data  rates  and  reliability  requirements  in 
war  vs.  peacetime  imply  a  new  or  novel  environment  for  the  system? 

•  What  consideration  does  the  acquisition  have  for  the  responsiveness  and 
timeliness  across  the  system?  If  there  is  a  requirement,  what  information  and 
activities  are  available  to  show  that  the  entire  suite  of  IT  (COTS  applications, 
networks,  servers,  and  so  forth)  will  meet  those  expectations?  If  there  are  no 
such  requirements,  how  will  the  installers  understand  and  judge  the  ability  to 
provide  a  system  that  the  users  will  find  acceptable? 

•  How  will  the  consistency  and  timeliness  of  data  be  ensured  by  the  selected 
suite  of  COTS  products?  Do  the  COTS  products  have  mechanisms  or  tech¬ 
niques  by  which  users  can  be  assured  that  they  have  the  latest  data  from  an 
authoritative  source?  How  will  the  authoritative  data  set  be  promulgated  and 
managed  across  the  system?  How  will  it  be  maintained  to  ensure  that  it  is 
updated  in  a  timely  fashion?  Does  the  system  have  enough  capacity  to  handle 
the  anticipated  data  storage  and  communication  requirements? 

•  How  do  issues  of  scalability  impact  the  selected  COTS  products?  Have  the 
products  been  run  in  organizations  that  have  similar  numbers  of  users,  similar 
sizes  of  data  sets,  and  similar  suites  of  applications?  Is  the  system  scalable  to 
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an  organization  commensurate  with  its  anticipated  use  in  DoD?  Is  that  seal- 
ability  affected  by  any  other  chosen  technologies  (e.g.,  IA)? 

•  Have  all  the  software  and  hardware  components  been  used  together  in  a  simi¬ 
lar  manner  with  similar  interfaces?  How  does  the  DoD  environment  differ 
from  the  environments  where  the  components  have  been  used  previously? 

•  Does  the  IA  requirement  imply  a  new  or  significantly  modified  security  envi¬ 
ronment?  Do  the  data  rates  and  reliability  requirements  in  war  vs.  peacetime 
imply  a  new  or  novel  environment  for  the  system?  Can  the  existing  network 
infrastructure  handle  the  anticipated  data  flow  requirements? 

D.4.8  Mission  Planning  Systems 

Mission  planning  systems  often  include  a  combination  of  COTS/GOTS  and 
developmental  software  to  integrate  software  systems.  For  these  systems,  usually  the 
components  are  mature  in  their  original  environment.  What  needs  to  be  determined  is 
how  the  new  integration  environment  differs.  Thus,  questions  might  include  the 
following: 

•  Are  there  new  logical  or  data  relationships  for  each  component?  Are  the 
algorithms  used  to  create  interfaces  new  or  novel?  Are  new  hardware  compo¬ 
nents  needed  to  enable  interoperability? 

•  Do  the  information  exchange  requirements  (IERs)  require  many  more  inter¬ 
faces  than  previously  achieved?  Does  this  imply  a  new  logical  or  security 
environment? 

•  Will  the  components  run  on  a  new  hardware  system?  On  a  new  network? 

•  Will  the  need  to  upgrade  the  components  introduce  new  algorithms  or 
technologies? 

D.4.9  Embedded  IT  in  Tactical  Systems 

Embedded  IT  or  software  in  tactical  systems  is  most  similar  to  developmental 
hardware.  Thus,  the  questions  would  include  the  following: 

•  Have  the  algorithms  been  proven  to  work  in  a  simulated  environment?  How 
is  that  environment  different  from  the  operational  environment? 

•  Does  the  data  dissemination  requirement  imply  a  new  or  novel  technology  or 
environment? 

•  Does  timeliness  imply  new  or  novel  algorithms  or  hardware?  Does  the  qual¬ 
ity  of  the  data  (e.g.,  engagement  quality)  imply  special  processing  that  has 
not  been  done  previously? 
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•  Are  the  number  of  software  systems  or  lines  of  code  unprecedented?  Do  the 
IERs  imply  a  new  or  novel  technology? 

D.4.10  Manufacturing 

The  following  questions  indicate  whether  a  manufacturing  technology  is  new  or 

novel: 

•  Has  the  manufacturing  technology  been  successfully  integrated  into  a 
product  line? 

•  Is  the  industrial  base85  capable  of  design,  development,  production,  mainte¬ 
nance  and  support,  and  disposal  of  the  system? 

•  Is  the  intended  design  producible? 

•  Have  the  materials  been  characterized  in  a  manufacturing  environment? 

•  Are  the  materials  available  to  meet  quantity  and  schedule  demands? 

•  Are  the  design-to-cost  (DTC)  goals  achievable? 

•  Are  the  key  manufacturing  processes  characterized,  capable,  and  controllable 
for  achieving  the  system  requirements? 


85  Depending  on  the  circumstances,  this  may  be  limited  to  the  National  industrial  base. 
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ACRONYMS  AND  ABBREVIATIONS  FOR  APPENDIX  D 


ATM 

Asynchronous  Transfer  Mode 

CAE 

Component  Acquisition  Executive 

Cl 

configuration  item 

COTS 

commercial  off-the-shelf 

CTE 

Critical  Technology  Element 

DAU 

Defense  Acquisition  University 

DoD 

Department  of  Defense 

DoDAF 

DoD  Architecture  Framework 

DTC 

design-to-cost 

FOR 

field  of  regard 

FOV 

field  of  view 

GOTS 

government  off-the-shelf 

HMMWV 

High  Mobility  Multipurpose  Wheeled  Vehicle 

IA 

Information  Assurance 

ICD 

Interface  Control  Document 

ICD 

Initial  Capabilities  Document 

IER 

information  exchange  requirement 

IT 

Information  Technology 

F/D 

lift-to-drag 

FAN 

local  area  network 

M&S 

modeling  and  simulation 

MAIS 

Major  Automated  Information  System 

MDAP 

Major  Defense  Acquisition  Program 

MIF-HDBK 

Military  Handbook 

NIC 

network  interface  card 

OSD 

Office  of  the  Secretary  of  Defense 

OV 

Operational  View 
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PDA 

personal  digital  assistant 

PM 

Program  Manager 

R&D 

research  and  development 

RAID 

redundant  array  of  inexpensive  disks 

S&T 

Science  and  Technology 

SAN 

storage  area  network 

SDD 

System  Design  and  Development 

SFC 

specific  fuel  consumption 

SUBSAFE 

Submarine  Safety  Certification  Program 

SV 

Systems  View 

TDA 

Technology  Development  Strategy 

TRL 

Technology  Readiness  Level 

TV 

Technical  Standards  View 

WAN 

wide  area  network 

WBS 

work  breakdown  structure 

WSERB 

Weapon  Systems  Explosive  Safety  Review  Board 
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APPENDIX  E. 

POLICY  STATEMENT 


National  Defense  Authorization  Act  for  Fiscal  Year  2002, 

107th  Congress,  House  of  Representatives,  Report  107-333, 

Conference  Report  To  Accompany  S.  1438,  December  12,  2001  . E-3 

SEC.  804.  Reports  on  Maturity  of  Technology  at  Initiation  of 

Major  Defense  Acquisition  Programs . E-4 
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107th  Congress  1  Report 

1st  Session  j  HOUSE  OF  REPRESENTATIVES  107_333 


NATIONAL  DEFENSE  AUTHORIZATION 
ACT  FOR  FISCAL  YEAR  2002 


CONFERENCE  REPORT 

TO  ACCOMPANY 

S.  1438 


December  12,  2001. — Ordered  to  be  printed 
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SEC.  804.  REPORTS  ON  MATURITY  OF  TECHNOLOGY  AT  INITIATION  OF 
MAJOR  DEFENSE  ACQUISITION  PROGRAMS. 

(a)  Reports  Required— Not  later  than  March  1  of  each  of 
years  2003  through  2006,  the  Secretary  of  Defense  shall  submit  to 
the  Committees  on  Armed  Services  of  the  Senate  and  the  House  of 
Representatives  a  report  on  the  implementation  of  the  requirement 
in  paragraph  4.7. 3. 2.2.2  of  Department  of  Defense  Instruction 
5000.2,  as  in  effect  on  the  date  of  enactment  of  this  Act,  that  tech¬ 
nology  must  have  been  demonstrated  in  a  relevant  environment  (or, 
preferably,  in  an  operational  environment)  to  be  considered  mature 
enough  to  use  for  product  development  in  systems  integration. 

(b)  CONTENTS  OF  Reports. — Each  report  required  by  subsection 
(a)  shall — 

(1)  identify  each  case  in  which  a  major  defense  acquisition 
program  entered  system  development  and  demonstration  during 
the  preceding  calendar  year  and  into  which  key  technology  has 
been  incorporated  that  does  not  meet  the  technological  maturity 
requirement  described  in  subsection  (a),  and  provide  a  justifica¬ 
tion  for  why  such  key  technology  was  incorporated;  and 

(2)  identify  any  determination  of  technological  maturity 
with  which  the  Deputy  Under  Secretary  of  Defense  for  Science 
and  Technology  did  not  concur  and  explain  how  the  issue  has 
been  or  will  be  resolved. 

(c)  Major  Defense  Acquisition  Program  Defined.— In  this 
section,  the  term  “major  defense  acquisition  program’’  has  the  mean¬ 
ing  given  that  term  in  section  139(a)(2)  of  title  10,  United  States 
Code. 
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APPENDIX  F. 

TECHNOLOGY  DEVELOPMENT  STRATEGY  (TDS)  TEMPLATE 


TDS  Template . F-3 

Acronyms  and  Abbreviations  for  Appendix  F  . F-5 
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Note:  The  Technology  Development  Strategy  (TDS)  is  a  precursor  to  the  Acquisition 
Strategy;  thus,  the  responsible  parties  for  oversight  are  part  of  Acquisition  Oversight,  not 
part  of  Science  and  Technology  (S&T)  Oversight.  The  template  is  included  here  as  ref¬ 
erence  because  the  TDS  is  an  important  prerequisite  to  the  Technology  Readiness  Asses¬ 
sment  (TRA). 


TDS  Template 

(Five  (5)  Page  Maximum) 

A.  Project/program  title:  (A  unique  title  specifically  identifying  this  proposed  project) 

B.  General  description  of  the  technology  solution:  Brief  overview  description  of  this 
technology  and  to  whom  it  will  provide  increased  capability  if  developed 

C.  Identify  the  development  strategy  (evolutionary  or  single-step-to-full-capability) 
and  provide  a  rationale  for  adopting  this  concept  and  technology  development 
approach.  For  evolutionary  strategy, 

1.  Describe  how  the  program  will  be  divided  into 

a.  Technology  spirals 

b.  Development  increments 

2.  Identify  an  appropriate  limitation  on  number  of  prototype  units  that  can  be 
produced 

3.  Describe  how  these  units  will  be  supported  (up  to  transition  to  the  customer) 

4.  Describe  the  specific  performance  goals  and  exit  criteria  that  must  be  met 
before  exceeding  the  number  of  prototypes 

D.  Project/program  strategy:  Describe  the  total  research  and  development  (R&D)  pro¬ 
gram  strategy,  including  all  spirals  and  so  forth,  to  include  the  following: 

1.  Overall  cost 

2.  Schedule 

3.  Performance  goals 
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E.  First  spiral  demonstration 

1.  Specific  cost 

a.  Development  cost:  estimate  from  project  start  to  transition  to  the 
customer) 

b.  Transition  and  integration  cost:  estimate  from  when  the  customer  receives 
the  project  for  integration  into  the  system  until  it  is  provided  to  the  user 

c.  Total  life-cycle  cost:  an  estimate  that  adds  operations  and  support  (O&S) 
and  disposal  costs  to  the  above 

2.  Schedule:  indicate  number  of  months  to  reach  each  Technology  Readiness 

Level  (TRL)  from  project  start  to  transition 

3.  Performance  goals  for  the  prototype  demonstration 

4.  Exit  criteria  for  the  prototype  demonstration  phase 

5.  Test  plan:  overview  concept  of  how  the  prototype  will  be  tested  and  how  the 

results  will  be  analyzed  for  Measures  of  Effectiveness  (MOEs) 

6.  Risk  strategy 

a.  Specify  the  technology  advancement  degree  of  difficulty  with  respect  to 
“state  of  the  art”  (1-well  within,  2-within,  3-pushing,  4-hard  push, 
5-breakthrough  required) 

b.  Identify  program  risks  for  the  first  spiral 

c.  Describe  mitigation  strategy 

7.  Transition  strategy:  an  overview  description  of  when,  to  whom,  and  under  what 

general  conditions  the  technology  solution  will  be  transitioned 
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ACRONYMS  AND  ABBREVIATIONS  FOR  APPENDIX  F 


MOE 

Measure  of  Effectiveness 

O&S 

operations  and  support 

R&D 

research  and  development 

S&T 

Science  and  Technology 

TDS 

Technology  Development  Strategy 

TRA 

Technology  Readiness  Assessment 

TRL 

Technology  Readiness  Level 
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APPENDIX  G. 

TECHNOLOGY  TRANSITION  AGREEMENT  (TTA) 
ELEMENTS  AND  TEMPLATE 


TTA  Elements  .  G-3 

Acronyms  and  Abbreviations  for  Appendix  G .  G-7 
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TTA  Elements 


A  TTA  documents  the  commitment  of  the  requirements/resource  sponsor,  the  sci¬ 
ence  and  technology  (S&T)  activity  (developer  and  provider  of  the  technology/product), 
and  the  Acquisition  Program  Office  (intended  receiver  of  a  technology  or  capability 
development)  to  develop,  deliver,  and  integrate  a  technology/product  into  an  acquisition 
program.  The  following  elements  should  be  considered  for  inclusion  in  the  TTA.  Not 
every  one  of  these  elements  is  appropriate  for  every  agreement,  but  each  element  should 
be  considered  for  inclusion. 

Agreements,  to  be  effective,  must  be  reviewed  periodically  with  each  of  the  key 
partners:  the  requirements/resource  sponsor,  S&T  activity  and  the  Acquisition  Program 
Office  representatives.  These  reviews  should  address  technical  progress  and  future 
directions. 

Elements  To  Be  Provided  by  the  Program  Office 

A.  Target  acquisition  program.  Provide  a  brief  description  of  the  acquisition  program  to 
receive  the  technology/product.  Include  the 

1 .  Maj  or  program  obj  ectives . 

2.  Current  phase  of  the  acquisition  life  cycle. 

3.  Projected  initial  operational  capability  date. 

B.  Program  Manager  (PM)/Project  Officer  (PO).  Identify  personnel  responsible  for  day- 
to-day  program/project  management. 

1.  PM  and  contact  information. 

2.  PO  and  contact  information. 

C.  Acquisition  program  technology  need.  Identify  the  technology  needs  of  the  acquisi¬ 
tion  program  that  S&T  is  expected  to  provide.  Briefly  describe  the  benefit  that  the 
technology/product  will  bring  to  the  acquisition  program. 

1.  Relate  the  benefit  to  the  Initial  Capabilities  Document  (ICD),  Capability  Devel¬ 
opment  Document  (CDD),  key  performance  parameters  (KPPs),  and  so  forth. 
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2. 


Include  need  dates  for  specific  capabilities. 

3.  Provide  an  estimate  of  the  Technology  Readiness  Level  (TRL)  for  each  technol¬ 
ogy/product  need  identified  using  a  systems  approach  for  hardware  and  soft¬ 
ware  as  the  measure  of  technical  maturity  and  as  an  indication  of  transition 
readiness.  Coordinate  the  TRL  with  the  S&T  activity. 

D.  Integration  strategy.  Describe  the  process  for  integrating  the  technology/product  into 

the  acquisition  program.  Include  the  following  elements  of  acquisition  strategy: 

1.  Evolutionary  acquisition,  block  upgrade,  and  so  forth. 

2.  Required  contractor-to-contractor  agreements. 

3.  Acquisition  program  element  (PE)  numbers  funding  the  transition. 

4.  Annual  PE  funding  levels  committed  to  the  transition  program. 

5.  Transition  Fiscal  Year  (FY). 

6.  Statement  conveying  the  level  of  commitment.  For  example, 

i.  Commitment:  “Upon  successful  demonstration  of  key  performance 
requirements  ( exit  criteria ),  PM  XXX  ( Acquisition  Program  Office)  will 
integrate  YYY  ( technology  product  delivered)  into  XXX  ( acquisition  pro¬ 
gram  that  will  integrate  the  technology  deliverable)  commencing  in  FYXX 
(transition  year).”  This  integration  effort  will  be  funded  under 
PE  XXXXXXX,  Project  XXXX  (FYDP  budget  profile  for  this  acquisition 
line  should  be  included). 

ii.  Intent:  “Upon  successful  demonstration  of  key  performance  requirements 
(exit  criteria),  PM  XXX  (Acquisition  Program  Office)  intends  to  integrate 
YYY  (technology  product  to  be  delivered)  into  XXX  (acquisition  program 
that  will  integrate  the  technology  deliverable)  commencing  in  FYXX 
(transition  year)  under  PE  XXXXXXX  Project  XXXX  (FYDP  budget 
profile).” 

Elements  To  Be  Provided  by  the  S&T  Activity 

A.  Development.  What  the  S&T  activity  intends  to  develop  for  transition  to  the  acquisi¬ 
tion  program.  Include  capability  delivery  dates. 
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B.  Technology  manager.  Identify  the  individual  designated  by  the  S&T  activity  to  coor¬ 
dinate  day-to-day  management  of  the  technology/product  development  and  list  con¬ 
tact  information. 

C.  Current  status  of  technology/product.  Show 

1.  Status  summary.  Summarize  current  state  of  development.  Identify: 

a.  Primary  areas  where  additional  development  is  required. 

b.  Estimate  of  current  TRL. 

2.  Risk  analysis.  Prioritize  and  discuss  major  areas  of  technical  risk.  Identify 
planned  mitigation  activities  to  address  technical  risk  (e.g.,  producibility, 
affordability,  sustainability). 

D.  Technology  Development  Strategy  (TDS).  Outline  planned  approach.  Include 

1.  Efforts  required  beyond  those  currently  underway. 

2.  Integration  plans  if  multiple  projects  are  planned. 

3.  Planned  Advanced  Technology  Demonstration  (ATD)  or  Advanced  Technology 
Concept  Demonstration  (ACTD)  developments,  if  applicable. 

E.  Exit  criteria  (key  technical  measures  of  readiness)  for  transition.  Identify  quantifi¬ 
able  criteria  that  will  be  used  to  measure  whether  the  technology/product  develop¬ 
ment  effort  is  proceeding  appropriately.  Provide 

1.  Definitive,  complete,  measurable  parameters  to  be  tracked,  to  include  perform¬ 
ance,  physical  attributes. 

2.  Conditions  under  which  technology/product  will  be  tested/demonstrated  before 
delivery  to  acquisition. 

3.  Current  performance  of  the  technology/product. 

4.  Minimum  acceptable  performance  threshold. 

5.  Desired  final  goal/objective. 

6.  Estimate  of  the  transition  TRL,  coordinated  with  the  program  office. 

F.  Program  plan.  Show  major  activities/efforts  planned  for  the  technology/product 
development,  with  milestones.  Include  both  S&T  and  acquisition  tasks/elements/ 
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Elements  To  Be  Provided  by  Resources/Requirements  Code 

A.  Capability  requirement  basis.  Identify  the  governing  source  of  the  capability  require¬ 
ment  (e.g.,  the  ICD,  CDD,  or  other  official  reference  documenting  the  capability 
need). 

B.  Resource  sponsor/requirements  officer.  Identify  the  resource  sponsor  and  require¬ 
ments  officer  responsible  for  resourcing  and  establishing  requirements  for  the  capa¬ 
bility.  Include  contact  information. 

Signatures  and  Dates 

TTAs  should  be  signed  to  commit  participating  organizations  to  the  plan  outlined  in 
the  agreement. 
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ACRONYMS  AND  ABBREVIATIONS  FOR  APPENDIX  G 


ACTD 

Advanced  Concept  Technology  Demonstration 

ATD 

Advanced  Technology  Demonstration 

CDD 

Capability  Development  Document 

FY 

Fiscal  Year 

FYDP 

Future  Years  Defense  Plan 

ICD 

Initial  Capabilities  Document 

KPP 

key  performance  parameter 

ONR 

Office  of  Naval  Research 

PE 

Program  Element 

PM 

Program  Manager 

PO 

Project  Officer 

S&T 

science  and  technology 

TDS 

Technology  Development  Strategy 

TRL 

Technology  Readiness  Level 

TTA 

Technology  Transition  Agreement 
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H.2  The  FDA  Regulatory  Process .  H-17 

H.2.1  Pharmaceuticals .  H-17 
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Biomedical 

—  Technology 

—  Readiness 
-  Levels  (TRLs) 


Note:  Medical-related  items  require  Technology  Readiness  Level  (TRL)  definitions  and 
descriptions  that  are  appropriate  to  the  technologies  upon  which  they  are  based  and  that 
account  for  the  statues  and  regulations  that  govern  their  development  and  use.  In  recog¬ 
nition  of  these  factors,  the  United  States  Army  Medical  Research  and  Materiel  Command 
(USAMRMC)  took  the  initiative  to  establish  appropriate  definitions,  descriptions,  and 
processes  in  the  context  of  military  medical  research  and  development  (R&D)  and  Food 
and  Drug  administration  (FDA)  statutory  and  regulatory  requirements.  This  appendix 
provides  the  results  of  their  effort. 

H.l  BACKGROUND 

Department  of  Defense  (DoD)  policy  mandates  the  use  of  U.S.  FDA-approved 
products  for  force  health  protection,86  and  the  USAMRMC  has  always  adhered  to  the 
regulatory  requirements  of  the  FDA  for  its  studies  of  drugs,  biologies,  and  devices  in 
humans.  To  ensure  compliance  with  the  clinical  phases  of  the  FDA-regulated  process  and 
to  reduce  technological  risk,  the  USAMRMC  developed  and  recently  updated  their  gen¬ 
eral  guidelines  for  assigning  TRLs  to  drug,  vaccine,  and  medical  device  development 
programs.87  These  guidelines  are  not  considered  absolutes,  and  characterization  of  activi¬ 
ties  associated  with  TRLs  can  and  does  vary  at  times. 

The  science  and  technology  (S&T)  and  acquisition  program  managers  (PMs) 
work  together  in  exercising  discretion  in  the  selection,  progression,  and  timing  of  specific 
activities  to  be  accomplished  in  the  attainment  of  particular  TRLs.  Such  flexibility  and 
tailoring  are  needed  to  align  the  TRL  decision  criteria  appropriately  with  the  maturation 
and  risk  characteristics  of  a  particular  technology,  including  consideration  of  the  associ¬ 
ated  investment  strategy  and  transition  procedures  that  may  vary  among  PMs. 


86  For  example,  Department  of  Defense  Directive  (DoDD)  6200.2,  Use  of  Investigational  New  Drugs  for 
Force  Health  Protection,  August  1,  2000,  or  Health  Affairs  Policy  95-011,  Tri-Service  Pharmacy  Pol¬ 
icy  Guidance,  July  26,  1995. 

87  Biomedical  Technology  Readiness  Levels  (TRLs),  prepared  for  the  Commander,  U.S.  Army  Medical 
Research  and  Materiel  Command,  under  Contract  DAMD17-98-D-0022,  Science  Applications 
International  Corporation,  3  June  2003. 
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When  transitioning  from  technology  development  to  product  development,  the 
risks  are  greater  if  the  TRL  of  a  Critical  Technology  Element  (CTE)  is  low.  For  medical 
technologies,  risk  reduction  is  not  linear  across  TRLs.  The  rate  of  risk  reduction  remains 
very  low  until  very  late.  Historically,  FDA-regulated  products,  such  as  vaccines,  do  not 
achieve  significant  risk  reduction  (i.e.,  greater  than  50  percent)  until  completion  of 
Phase  3  clinical  trials  and  approval  of  a  biologies  license  application  by  the  FDA 
(TRF  8).  Industry’s  experience  is  that  only  one  in  four  vaccines  going  into  Phase  3  trials 
is  licensed.  Similarly,  whereas  technology  maturation  is  commonly  perceived  as  a 
sequential  continuum  of  activities  from  basic  research,  through  development,  to  produc¬ 
tion  and  deployment,  the  evolution  of  the  TRF  for  a  biomedical  CTE  may  not  be  sequen¬ 
tial,  especially  in  those  cases  where  FDA  anchors  are  undefined.  In  cases  of  success  or 
failure,  the  incremental  change  in  the  level  of  technology  readiness  may  be  greater  than  a 
single  TRF.  For  example,  upon  successful  completion  of  a  pivotal  study,  biomedical 
information  readiness  levels  may  move  from  TRF  3  or  4  to  TRF  9. 

Biomedical  TRF  descriptions  provide  a  systematic  way  for  the  S&T  community 
to  assess  and  communicate  to  the  Milestone  Decision  Authority  (MDA)  the  maturity 
level  of  a  particular  technology  or  combination  of  technologies  and  the  maturity  neces¬ 
sary  for  successful  product  development.  This  appendix  provides  equivalent  TRF 
descriptions  applicable  to  biomedical  technologies  in  four  categories: 

1.  Pharmaceutical  (i.e.,  drugs) 

2.  Pharmaceutical  (i.e.,  biologics/vaccines) 

3.  Medical  Devices 

4.  Medical  Information  Management/Information  Technology  (IM/IT)  and 
Medical  Informatics. 

The  TRFs  for  the  first  three  categories  have  been  developed  from  the  DoD’s 
generic  definitions,  the  applicable  FDA  regulatory  process,  and  industry  practices  and 
experience  with  its  R&D  processes  (discovery  through  manufacturing,  production,  and 
marketing).  The  last  category  includes  elements  of  formal  regulatory  processes  and  logi¬ 
cal  events  in  deriving  comparable  levels  of  maturity.  Wherever  practical,  the 
USAMRMC  intends  to  use  external  anchors  such  as  “FDA  events”  to  define  each  TRF 
decision  criterion.  Furthermore,  activities  described  as  occurring  between  successive 
TRF  decision  criteria  are  intended  to  exemplify  the  kinds  of  activities  that  routinely  take 
place  when  maturation  is  sequential  and  stepwise.  However,  these  examples  are  neither 
mandatory  nor  all-inclusive. 

Figure  H-l  and  Table  H-l  build  upon  this  work  by  providing  examples  of  sup¬ 
porting  information  and  documentation  required  to  support  the  assignment  of  TRFs  as  the 
program  progresses. 

The  proponent  for  this  document  is  the  Deputy  for  Research  and  Development: 
Commander,  U.S.  Army  Medical  Research  and  Materiel  Command,  ATTN: 
MCMR-ZC,  504  Scott  Street,  Fort  Detrick,  MD  21702-5012. 


H-4 


Technology  Readiness  Assessment 
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Figure  H-1.  TRLs  in  the  Medical  Materiel  Regulatory  Process 

Note  for  Figure  H-1 :  The  TRL  descriptions  are  not  considered  absolutes,  and  characterization  of  activities  associated  with  TRLs  can  and  does  vary  at 
times.  The  S&  T  and  acquisition  PMs  work  together  in  exercising  discretion  in  the  selection,  progression,  and  timing  of  specific  activities  to  be  accom¬ 
plished,  particularly  with  regard  to  TRL  5.  Such  flexibility  and  tailoring  are  needed  to  align  the  TRL  decision  criteria  appropriately  with  maturation  and 
risk  characteristics  of  a  particular  technology,  including  consideration  of  the  associated  investment  strategy  and  transition  procedures  that  may  vary 
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The  Defense  Acquisition  Guidebook,  dated  October  2004,  contains  nonmandatory  guidance  on  best  practices,  lessons  learned,  and  expectations. 
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A  510(k)  is  a  premarket  notification  for  medical  devices. 
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H.2  THE  FDA  REGULATORY  PROCESS 


To  protect  U.S.  public  health,  the  FDA  regulates  products  by  ensuring  that  human  pharmaceu¬ 
ticals  (drugs  and  biologics/vaccines)  are  safe  and  effective  and  that  reasonable  assurance 
exists  concerning  the  safety  and  effectiveness  of  medical  devices  intended  for  human  use. 
Three  FDA  centers  are  charged  with  this  mission: 

1.  The  Center  for  Drug  Evaluation  and  Research  (CDER).  CDER  regulates  drugs  and 
some  biologic  products  (antibodies,  cytokines,  growth  factors,  enzymes,  and  proteins 
extracted  from  animals  or  microorganisms). 

2.  The  Center  for  Biologies  Evaluation  and  Research  (CBER).  CBER  regulates  vac¬ 
cines,  blood  and  plasma  products,  viral- vectored  gene  therapy,  products  composed  of 
human  or  animal  cells,  antitoxins,  and  select  in  vitro  diagnostics.  CBER  also  holds 
regulatory  authority  over  Human  Immunodeficiency  Virus  (HIV)  test  kits  and  medical 
devices  involved  in  collecting,  processing,  testing,  manufacturing,  and  administering 
blood  products. 

3.  The  Center  for  Devices  and  Radiological  Health  (CDRH).  CDRH  is  responsible  for 
regulating  manufactured,  repackaged,  relabeled,  and/or  imported  medical  devices  that 
are  sold  in  the  United  States  (except  those  devices  regulated  by  CBER). 

H.2.1  Pharmaceuticals 

Drugs  and  biologics/vaccines  follow  parallel  developmental  regulatory  pathways  (see 
Table  H-l).  During  preclinical  development,  the  sponsor  evaluates  the  toxicology  and  phar¬ 
macology  of  the  new  drug  or  biologic  through  in  vitro  and  animal  testing.  Preclinical  test 
results  and  any  available  past  human  experiences  of  the  drug  or  biologic  are  incorporated  in 
an  IND  (Investigational  New  Drug  Application)  and  submitted  to  the  FDA  for  review.  If  no 
safety  issues  are  found,  human  clinical  testing  of  the  new  drug  or  biologic  can  be  initiated 
after  30  days.  Clinical  testing  proceeds  in  three  successive  phases,  starting  with  a  small  group 
of  human  subjects  (Phase  1)  and  progressing  to  a  larger  population  of  human  subjects 
(Phase  3).  Only  qualified  investigators,  selected  by  the  sponsor  in  accordance  with  GCP 
(Good  Clinical  Practice)  (21CFR312.53  and  21CFR312.62),  conduct  clinical  trials.  The  safety 
and  effectiveness  results  of  clinical  testing  comprise  the  most  important  factor  in  the  approval 
or  disapproval  of  the  new  drug  or  biologic.  All  active  INDs  require  submission  of  an  annual 
IND  report  to  the  FDA.  The  results  of  the  human  clinical  tests  and  all  chemistry  and  manufac¬ 
turing  information  are  submitted  either  in  an  NDA  (New  Drug  Application)  for  drug  products 
or  a  BLA  (Biologies  License  Application)  for  biologic  products.  The  appropriate  FDA  center 
reviews  the  NDA  or  BLA,  and,  upon  approval,  the  drug  or  biologic  product  can  be  entered 
into  interstate  commerce  or  marketed  in  the  United  States.  FDA  approval  is  for  the  specific 
indication(s)  identified  in  the  marketing  application.  Additional  or  modified  medical  indica¬ 
tions  require  the  submission  of  an  amendment  or  a  new  marketing  application.  A  new  mar¬ 
keting  application  may  require  additional  human  clinical  data  acquired  through  IND 
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regulations.  With  some  new  drugs  or  biologic  s/vaccines,  the  FDA  may  require  additional 
reporting  requirements  after  approval,  termed  Phase  4  or  postmarketing  surveillance.  Manu¬ 
facturers  are  required  to  track  and  report  the  number  and  severity  of  adverse  events  attribut¬ 
able  to  each  product  for  a  specified  time  period.  Severe  adverse  events  detected  during 
postapproval  can  lead  to  a  product  recall  or  mandatory  withdrawal  from  the  market.  All  drugs 
and  biologics/vaccines  must  comply  with  cGMP  (current  Good  Manufacturing  Practice)  and 
labeling  regulations. 

With  certain  drugs  or  biologic  products,  human  clinical  studies  are  not  ethical  or  feasi¬ 
ble  because  the  studies  would  involve  administering  a  potentially  lethal  or  permanently  dis¬ 
abling  toxic  substance  or  organism  to  healthy  human  volunteers.  In  2002,  the  FDA  addressed 
this  issue  with  new  regulations  that  allow  for  the  approval  of  new  drug  and  biologic  products 
based  on  evidence  of  effectiveness  in  animals  (21CFR314  and  21CFR601).  In  February  2003, 
under  the  new  federal  regulations,  DoD  was  able  to  gain  approval  of  pyridostigmine  bromide 
for  prophylaxis  against  the  lethal  effects  of  the  soman  nerve  agent. 

H.2.2  Medical  Devices 

The  FDA  CDRH  regulates  most  medical  devices,  and  they  have  classified  each  device 
in  the  Code  of  Federal  Regulations  (CFR).  Classification  of  devices  into  one  of  three  classes 
is  based  on  the  level  of  regulatory  control  that  is  necessary  to  ensure  the  safety  and  effective¬ 
ness  of  a  medical  device,  with  Class  I  and  Class  III  devices  being  the  least  and  most  regulated, 
respectively.  The  sponsor  normally  proposes  the  classification  level  of  a  device  using 
21CFR860  as  a  guide.  Most  importantly,  the  classification  of  the  device  will  identify,  unless 
exempt  (e.g.,  most  of  the  Class  I  devices),  the  marketing  process  [either  premarket  notification 
(510(k))  or  PMA  (Premarket  Approval)]  that  the  manufacturer  must  complete  to  obtain  FDA 
clearance/approval  for  marketing.  All  classified  medical  devices  are  subject  to  cGMP  and 
labeling  requirements.  An  approved  510(k)  or  PMA  allows  an  applicant  to  market  a  particular 
device  for  its  intended  purpose. 

The  FDA  approves  most  medical  devices  for  marketing  in  the  United  States  through  a 
premarket  notification  [510(k)].  The  applicant  must  show  that  the  new  device  is  substantially 
equivalent  to  one  or  more  predicate  devices  legally  marketed  in  the  United  States.  A  descrip¬ 
tion  of  all  tests  conducted  and  the  results  obtained  must  be  provided  in  sufficient  detail  to 
allow  the  FDA  to  determine  substantial  equivalence.  If  the  medical  device  is  found  to  be  sub¬ 
stantially  equivalent,  the  FDA  will  send  the  manufacturer  a  “substantially  equivalent  letter”  to 
clear  the  device  for  marketing.  If  the  FDA  finds  the  device  not  to  be  substantially  equivalent, 
the  FDA  sends  the  manufacturer  a  “not  substantially  equivalent  letter,”  and  the  device  cannot 
be  marketed.  At  this  point,  the  manufacturer  can  submit  another  510(k)  with  new  and/or  addi¬ 
tional  information  to  support  substantial  equivalence  or  may  be  required  to  submit  a  PMA. 

To  allow  a  Class  III  medical  device  (devices  that  support  or  sustain  human  life  or  pre¬ 
sent  a  potential  risk  of  serious  illness  or  injury)  into  interstate  commerce  or  marketing,  a  PMA 
is  required.  A  PMA  is  the  most  stringent  regulatory  submission  for  medical  devices.  Class  III 
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devices  follow  somewhat  different  development  and  regulatory  paths  compared  with  those  for 
drugs  and  biologics/vaccines  (see  Table  H-l).  For  example,  if  human  clinical  information  is 
required  to  establish  safety  and  efficacy,  the  regulatory  application  that  allows  human  clinical 
trials  is  called  an  IDE  (Investigational  Device  Exemption).  Approval  of  an  IDE  allows  the 
initiation  of  human  clinical  trials  of  an  investigational  device.  Qualified  principal  investigators 
(Pis),  selected  by  the  sponsor  in  accordance  with  21CFR812.43,  conduct  clinical  trials.  All 
active  IDEs  require  submission  of  an  annual  report  to  the  FDA.  Safety  and  efficacy  informa¬ 
tion  acquired  during  the  IDE  process  is  used  to  support  the  submission  of  a  PMA,  and  the 
FDA  must  approve  the  PMA  before  the  device  can  be  marketed.  As  with  drugs  and  biol¬ 
ogics/vaccines,  the  FDA  may  mandate  a  period  of  postmarketing  surveillance  during  which 
device-related  adverse  events  must  be  tracked  and  reported. 

H.3  WEB  SITES 

FDA  Center  for  Devices  and  Radiological  Health  (CDRH): 

http://www.fda.gov/cdrh/ 

FDA  Center  for  Drug  Evaluation  and  Research  (CDER): 

http://www.fda.gov/cder/ 

FDA  Center  for  Biologies  Evaluation  and  Research  (CBER): 

http://www.fda.gov/cber/ 

H.4  ADDITIONAL  INFORMATION 

Federal  Food,  Drug,  and  Cosmetic  (FD&C)  Act 

United  States  Code,  Title  21  -  Food  and  Drugs  (21USC) 

Chapter  9:  Federal  Food,  Drug,  and  Cosmetic  Act 
http  ://w  w  w .  access,  gpo .  go  v/uscode/title2 1  /chapter9_.html 

FDA  Regulations 

CFR:  Title  21  -  Food  and  Drugs  (21CFR) 

http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfCFR/CFRSearch.cfm  or 
http  ://w  w  w .  gpoaccess .  go  v/cfr/index.html 

Drug  Approval 

The  CDER  Handbook :  http://www.fda.gov/cder/handbook/ 

CDERLearn:  http://www.fda.gov/cder/learn/CDERFearn/default.htm 

Medical  Device  Approval 

Device  Advice:  http://www.fda.gov/cdrh/devadvice/index.html 
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Laws  Enforced  by  the  FDA 

http://www.fda.gov/opacom/laws/ 

Protection  of  Human  Subjects 

32CFR219-  Protection  of  Human  Subjects  (also  referred  to  as  the  “Common  Rule”) 
(http://www.access.gpo.gov/nara/cfr/waisidx_02/32cfr219_02.html) 

DoDD  3216.2  (March  25,  2002)  Protection  of  Human  Subjects  and  Adherence  to  Ethical 
Standards  in  DoD-Supported  Research 

http://www.dtic.mil/whs/directives/corres/pdf/d32162_032502/d32162p.pdf 
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GLOSSARY  FOR  APPENDIX  H90 


Approval  Letter:  A  written  communication  to  an  applicant  from  the  FDA  approving  an 
application  or  an  abbreviated  application  to  market  a  drug.  [21CFR314.3] 

Approval  Order:  A  written  communication  to  an  applicant  from  the  FDA  approving  a  PMA 
for  a  Medical  Devices  application.  [21CFR814.44] 

Biologic  or  Biological  Product:  Any  virus,  therapeutic  serum,  toxin,  antitoxin,  or  analogous 
product  applicable  to  the  prevention,  treatment,  or  cure  of  diseases  or  injuries  of  man. 
[21CFR600.3] 

Biologies  License  Application  (BLA):  An  application  to  the  FDA  for  approval  to  market  a 
biological  product.  [21CFR601.12] 

current  Good  Manufacturing  Practice  (cGMP):  Regulations  that  cover  the  methods  used  in 
and  the  facilities  and  controls  used  for  the  design,  manufacture,  packaging,  storage,  and 
installation  of  devices.  [21CFR820] 

Class  (Device):  One  of  the  three  categories  of  regulatory  control  for  medical  devices. 
[21CFR860.3] 

Class  I  Device:  The  class  of  devices  for  which  general  controls  are  sufficient  to  provide  rea¬ 
sonable  assurance  of  the  safety  and  effectiveness  of  the  device.  In  the  absence  of  sufficient 
information  to  make  that  determination,  the  device  is  not  life  supporting  and  does  not  present 
a  potential  unreasonable  risk  of  illness  or  injury.  [21CFR860.3] 

Class  II  Device:  The  class  of  devices  for  which  general  controls  alone  are  insufficient  to  pro¬ 
vide  reasonable  assurance  of  its  safety  and  effectiveness  and  for  which  there  is  sufficient 
information  to  establish  special  controls,  including  the  promulgation  of  performance  stan¬ 
dards.  For  a  device  that  is  purported  to  be  for  use  in  supporting  human  life,  the  Commissioner 
(FDA)  shall  examine  and  identify  the  special  controls,  if  any,  that  are  necessary  to  provide 
adequate  assurance  of  safety  and  effectiveness.  [21CFR860.3] 


90  Complete  definitions  and  explanations  of  terms  can  be  found  in  the  source  cited  in  brackets.  CFR  is  an 
acronym  for  the  Code  of  Federal  Regulations. 
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Class  III  Device:  The  class  of  devices  for  which  premarket  approval  is  or  will  be  required.  A 
device  is  in  Class  III  if  insufficient  information  exists  to  determine  that  general  controls  are 
sufficient  to  provide  reasonable  assurance  of  its  safety  and  effectiveness,  if  the  device  is  life 
supporting,  or  if  the  device  presents  a  potential  unreasonable  risk  of  illness  or  injury. 
[21CFR860.3] 

Classification  Name:  The  term  used  by  the  FDA  and  its  classification  panels  to  describe  a 
device  or  class  of  devices  for  purposes  of  classifying  devices  under  section  513  of  the  Federal 
Food,  Drug,  and  Cosmetic  (FD&C)  Act.  [21CFR807.3] 

Approximately  1,700  different  generic  types  of  devices  are  grouped  into  16  medical  special¬ 
ties  [21CFR862-892],  as  follows: 

862:  Clinical  chemistry  and  clinical  toxicology  devices 

864:  Hematology  and  pathology  devices 

866:  Immunology  and  microbiology  devices 

868:  Anesthesiology  devices 

870:  Cardiovascular  devices 

872:  Dental  devices 

874:  Ear,  nose  and  throat  devices 

876:  Gastroenterology-urology  devices 

878:  General  and  plastic  surgery  devices 

880:  General  hospital  and  personal  use  devices 

882:  Neurological  devices 

884:  Obstetrical  and  gynecological  devices 

886:  Ophthalmic  devices 

888:  Orthopedic  devices 

890:  Radiology  devices 

892:  Banned  devices. 

Clinical  Hold:  An  FDA  order  to  delay  proposed  clinical  investigation  or  to  suspend  an  on¬ 
going  investigation. 

Clinical  Investigation:  Any  experiment  in  which  a  drug  that  involves  one  or  more  human 
subjects  is  administered,  dispensed,  or  used.  For  this  part,  an  experiment  is  any  use  of  a  drug 
except  for  the  use  of  a  marketed  drug  in  the  course  of  medical  practice.  [21CFR312.3] 

Clinical  Trial/Clinical  Study:  Any  investigation  in  human  subjects  intended  to  discover  or 
verify  the  clinical,  pharmacological,  and/or  other  pharmacodynamic  effects  of  an  investiga¬ 
tional  product(s),  and/or  identify  any  adverse  reactions  to  an  investigational  product(s),  and/or 
study  absorption,  distribution,  metabolism,  and  excretion  of  an  investigational  product(s)  with 
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the  object  of  ascertaining  its  safety  and/or  efficacy.  The  terms  clinical  trial  and  clinical  study 
are  synonymous.  [62  FR  25692]91 

Cosmetic:  (1)  Articles  intended  to  be  rubbed,  poured,  sprinkled  or  sprayed  on,  or  introduced 
into  or  otherwise  applied  to  the  human  body  or  any  part  thereof  for  cleansing,  beautifying, 
promoting  attractiveness,  or  altering  appearance  and  (2)  articles  intended  for  use  as  a  compo¬ 
nent  of  any  such  article.  This  term  shall  not  include  soap. 

Device  Master  Record  (DMR):  A  compilation  of  records  containing  the  procedures  and 
specifications  for  a  finished  device.  [21CFR820.3] 

Drug  or  Drug  Substance:  An  active  ingredient  that  is  intended  to  furnish  pharmacological 
activity  or  other  direct  effect  in  the  diagnosis,  cure,  mitigation,  treatment,  or  prevention  of 
disease  or  to  affect  the  structure  or  any  function  of  the  human  body.  [21CFR314.3] 

Drug  Product:  A  finished  dosage  form  (e.g.,  tablet,  capsule,  or  solution)  that  contains  a  drug 
substance,  generally,  but  not  necessarily,  in  association  with  one  or  more  other  ingredients. 
[21CFR314.3] 

FD&C  Act:  The  Federal  Food,  Drug,  and  Cosmetic  Act.  [21USC301-397] 

FDA-Approved:  An  FDA  designation  given  to  drugs,  biologies,  and  medical  devices  that 
have  approved  marketing  applications.  Additional  or  modified  medical  indications  for  use 
require  the  submission  of  an  amendment  or  a  new  marketing  application.  A  new  marketing 
application  may  require  additional  human  clinical  data  acquired  through  IND  regulations. 

General  Controls:  The  baseline  requirements  of  the  FD&C  Act  that  apply  to  all  medical 
devices.  In  addition  to  prohibiting  adulteration,  misbranding,  and  banned  devices,  the  general 
controls  contain  requirements  for  device  manufacturers.  These  requirements  include  device 
listing,  proper  labeling,  (manufacturing)  establishment  registration,  and  premarket  notification 
[5 10(k)]. 

Good  Clinical  Practice  (GCP):  A  standard  for  the  design,  conduct,  performance,  monitoring, 
auditing,  recording,  analyses,  and  reporting  of  clinical  trials.  It  provides  assurance  that  the 
data  and  reported  results  are  credible  and  accurate  and  that  the  rights,  integrity,  and  confiden¬ 
tiality  of  trial  subjects  are  protected.  [62  FR  25692] 


91  62  FR  25692  (May  9,  1997)  is  an  International  Conference  on  Harmonisation  (ICH)  document  called  Good 

Clinical  Practice:  Consolidated  Guideline.  This  document  addresses  GCP  principles  that  were  adopted  for 
use  as  guidance  for  industry.  ICH  is  a  joint  initiative  involving  both  regulators  and  industry  as  equal  partners 
in  the  scientific  and  technical  discussions  of  the  testing  procedures  that  are  required  to  ensure  and  assess  the 
safety,  quality  and  efficacy  of  medicines. 
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Good  Laboratory  Practice  (GLP):  Practices  for  conducting  nonclinical  laboratory  studies 
that  support  or  are  intended  to  support  applications  for  research  or  marketing  permits  for 
products  regulated  by  the  FDA.  [21CFR58.1] 

Investigational  Device  Exemption  (IDE):  Allows  the  investigational  device  to  be  used  in  a 
clinical  study  to  collect  safety  and  effectiveness  data  required  to  support  a  PMA  application  or 
a  Premarket  Notification  [510(k)]  submission  to  the  FDA.  [21CFR50,  56,  812] 

Investigational  New  Drug  (IND):  A  new  drug  or  biologic  that  is  used  in  a  clinical  investi¬ 
gation.  The  term  also  includes  a  biological  product  that  is  used  in  vitro  for  diagnostic  pur¬ 
poses.  [21CFR312.3] 

IND  Application:  Allows  a  pharmaceutical  (drug/biologic)  to  be  used  in  a  study  under  care¬ 
fully  controlled  and  intensely  monitored  conditions  in  order  to  collect  safety  and  effectiveness 
data  required  to  support  an  NDA  or  BLA.  [21CFR312.3] 

Investigator:  A  person  responsible  for  the  conduct  of  the  clinical  trial  at  a  trial  site.  If  a  trial 
is  conducted  by  a  team  of  individuals  at  a  trial  site,  the  investigator  is  the  responsible  leader  of 
the  team  and  may  be  called  the  principal  investigator  (or  PI).  [62  FR  25692] 

Label:  Any  display  of  written,  printed,  or  graphic  matter  on  the  immediate  container  or  pack¬ 
age  of,  or  affixed  to  any  article. 

Labeling:  Any  written,  printed,  or  graphic  matter  accompanying  an  article  at  any  time  while 
such  article  is  in  interstate  commerce  or  held  for  sale  after  shipment  in  interstate  commerce. 
This  includes  manuals,  brochures,  advertising,  and  so  forth. 

License:  The  terminology  used  for  FDA’s  approval  to  market  a  biological  pharmaceutical  for 
a  given  set  of  indications  (see  also  FDA  Approved). 

Life-Supporting  or  Life-Sustaining  Device:  A  device  that  is  essential  to  or  that  yields 
information  that  is  essential  to  the  restoration  or  continuation  of  a  bodily  function  important  to 
the  continuation  of  human  life.  [21CFR860.3] 

Medical  Device:  An  instrument,  apparatus,  implement,  machine,  contrivance,  implant,  in 
vitro  reagent,  or  other  similar  or  related  article,  including  any  component,  part,  or  accessory 
that  is 

•  Recognized  in  the  official  National  Formulary  or  U.S.  Pharmacopoeia  or  any  supple¬ 
ment  to  them 

•  Intended  for  use  in  diagnosing  disease  or  other  conditions  or  in  curing,  mitigating, 
treating,  or  preventing  disease  in  man  or  other  animals 

•  Intended  to  affect  the  structure  or  any  function  of  the  body  of  man  or  other  animals 
and  does  not  achieve  any  of  its  primary  intended  purposes  through  chemical  action 
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within  or  on  the  body  of  man  or  other  animals  and  is  not  dependent  upon  being 
metabolized  for  achievement  of  any  of  its  primary  intended  purposes  [Section  201(h) 
of  the  FD&C  Act]. 

New  Drug  Application  (NDA):  An  application  to  the  FDA  for  approval  to  market  a  new 
drug.  [21CFR314.50] 

Preapproval  Inspection  (PAI):  An  FDA  inspection  of  a  facility  to 

•  Verify  the  integrity  (truthfulness,  accuracy,  and  completeness)  of  data  submitted  in 
support  of  an  application 

•  Evaluate  the  manufacturing  controls  for  the  preapproval  batches  upon  which  the  appli¬ 
cation  is  based  to  be  certain  that  the  company  can  actually  meet  the  commitments  in 
the  chemistry,  manufacturing,  and  controls  (CMC)  section  of  the  application 

•  Evaluate  the  capability  of  the  manufacturer  to  comply  with  GMPs 

•  Collect  samples  for  analysis. 

Postmarketing  Surveillance:  Tracking  and  reporting  the  number  and  severity  of  adverse 
events  attributable  to  each  product.  This  may  be  a  requirement  for  licensure  for  a  defined 
period  of  time  following  licensure. 

Premarket  Approval  (PMA)  for  Medical  Devices:  Because  of  the  level  of  risk  associated 
with  Class  III  devices,  an  applicant  must  receive  FDA  approval  of  its  PMA  application  before 
marketing  the  device.  PMA  approval  is  based  on  the  FDA’s  determination  that  the  PMA  con¬ 
tains  sufficient  valid  scientific  evidence  to  ensure  that  the  device  is  safe  and  effective  for  its 
intended  use(s).  [21CFR814] 

Premarket  Notification  [510(k)]:  An  application  submitted  to  the  FDA  to  demonstrate  that  a 
device  is  substantially  equivalent  [see  21USC513(I)(1)(A)]  to  a  device  that  is  legally  in  com¬ 
mercial  distribution  in  the  United  States  before  May  28,  1976,  or  to  a  device  that  has  been 
determined  by  FDA  to  be  substantially  equivalent.  [21CFR807.81] 

Quality  System  Inspection  Technique  (QSIT):  An  FDA  inspection  technique  that  focuses 
on  the  first  four  elements  of  the  seven  inspectional  subsets  of  the  Quality  System  Regulation 

(QSR). 

Quality  System  Regulation  (QSR):  The  1996  rewrite  of  the  device  section  of  the  cGMPs. 
[21CFR820] 

Serious  Adverse  Event  (SAE)  or  Serious  Adverse  Drug  Reaction  (ADR):  Any  untoward 
medical  occurrence  that  at  any  dose 

•  Results  in  death 

•  Is  life  threatening 
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Requires  inpatient  hospitalization  or  prolongation  of  existing  hospitalization 
Results  in  persistent  or  significant  disability /incapacity 
Causes  a  congenital  anomaly/birth  defect. 


Special  Controls:  Class  II  devices  include  any  device  for  which  reasonable  assurance  of 
safety  and  effectiveness  can  be  obtained  by  applying  “special  controls.”  Special  controls  can 
include  special  labeling  requirements,  mandatory  performance  standards,  patient  registries, 
and  postmarket  surveillance. 

Sponsor:  An  individual,  company,  institution,  or  organization  that  takes  responsibility  for  the 
initiation,  management,  and/or  financing  of  a  clinical  trial.  [62  FR  25692] 

Subject:  A  human  who  participates  in  an  investigation,  either  as  a  recipient  of  the  IND  or  as  a 
control.  [21CFR312.3] 

Substantial  Equivalence  (SE):  A  device  is  substantially  equivalent  if,  in  comparison  to  a 
legally  marketed  device,  it  has  the  same  intended  use  as  a  predicate  device  and  has  the  same 
technological  characteristics  as  the  predicate  device.  SE  does  not  mean  the  devices  are  identi¬ 
cal.  [21CFR807.87] 

Type  B  Meeting:  Type  B  meetings  are  (1)  pre-IND  meetings  (21CFR312.82),  (2)  certain  end 
of  Phase  1  meetings  (21CFR312.82),  (3)  end  of  Phase  2/pre-Phase  3  meetings 
(21CFR3 12.47),  and  (4)  pre-NDA/BLA  meetings  (21CFR312.47). 
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ACRONYMS  AND  ABBREVIATIONS  FOR  APPENDIX  H 


510(k) 

Premarket  Notification  for  Medical  Devices 

ADR 

Adverse  Drug  Reaction 

BLA 

Biologies  License  Application 

CBER 

Center  for  Biologies  Evaluation  and  Research 

CDER 

Center  for  Drug  Evaluation  and  Research 

CDRH 

Center  for  Devices  and  Radiologic  Health 

CFR 

Code  of  Federal  Regulations 

cGMP 

current  Good  Manufacturing  Practice 

CMC 

chemistry,  manufacturing,  and  controls 

CTE 

Critical  Technology  Element 

DMR 

Device  Master  Record 

DoD 

Department  of  Defense 

FD&C 

Federal  Food,  Drug,  and  Cosmetic 

FDA 

Food  and  Drug  Administration 

GCP 

Good  Clinical  Practice 

GLP 

Good  Laboratory  Practice 

HIV 

Human  Immunodeficiency  Virus 

HW/SW 

hardware/software 

ICH 

International  Conference  on  Harmonisation 

IDE 

Investigational  Device  Exemption 

IM/IT 

Information  Management/Information  Technology 

IND 

Investigational  New  Drug  Application 

IOT&E 

initial  operational  test  and  evaluation 

MDA 

Milestone  Decision  Authority 

NASA 

National  Aeronautics  and  Space  Administration 

NDA 

New  Drug  Application 
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PAI 

Preapproval  Inspection 

PI 

principal  investigator 

PM 

program  manager 

PMA 

Premarket  Approval 

POC 

point  of  contact 

QSIT 

Quality  System  Inspection  Technique 

QSR 

Quality  System  Regulation 

R&D 

research  and  development 

RDT&E 

research,  development,  test  and  evaluation 

S&T 

science  and  technology 

SAE 

Serious  Adverse  Event 

SE 

Substantial  Equivalence 

T&E 

test  and  evaluation 

TRL 

Technology  Readiness  Level 

use 

United  States  Code 

USAMRMC 

United  States  Army  Medical  Research  and  Materiel  Command 
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1.1.  BACKGROUND 


Manufacturing  readiness  and  producibility  are  as  important  to  the  successful 
development  of  a  system  as  are  readiness  and  the  capabilities  of  the  technologies 
intended  for  the  system.  Their  importance  has  long  been  recognized  in  Department  of 
Defense  (DoD)  acquisition.  They  remain  central  to  achieving  the  Secretary  of  Defense’s 
transformational  goals  to  realign  support  to  the  warfighter  by  reducing  acquisition  cycle 
times  and  eliminating  cost  growth.92 

DoD  Manufacturing  Readiness  Levels  (MRLs)  are  designed  to  be  measures  used 
to  assess  maturity  from  a  manufacturing  perspective.  The  purpose  of  MRLs  is  to  provide 
decision-makers  (at  all  levels)  a  common  understanding  of  the  relative  maturity  (and 
attendant  risks)  associated  with  manufacturing  technologies,  products,  and  processes 
being  considered  to  meet  DoD  requirements. 

A  Transition  Working  Group,  comprised  of  representatives  from  the  Military 
Services,  Defense  Logistics  Agency  (DLA),  Missile  Defense  Agency  (MDA),  and 
industry,  recently  addressed  the  issue  of  the  rapid,  affordable  transition  of  technology  to 
acquisition.  This  group  also  generated  an  initial  set  of  definitions  and  descriptions  for 
MRLs.  Subsequently,  the  Joint  Defense  Manufacturing  Technology  Panel  (JDMTP)93 
chartered  a  working  group  to  refine  the  initial  set  of  MRL  definitions  and  descriptions,  to 
deploy  MRLs  within  existing  DoD  acquisition  doctrine,  policy  guidance,  and  functional 
critical  processes,  and  to  institutionalize  MRLs  within  the  DoD  Acquisition,  Technology, 
and  Logistics  (AT&L)  community. 


92  2003  Secretary  of  Defense  Annual  Report  to  the  President  and  the  Congress ,  pp.  60-62. 

93  The  JDMTP  vision  is  to  rapidly  transition  science  and  technology  (S&T)  from  discovery  and  inven¬ 
tion,  through  engineering  development,  onto  the  factory  floor,  and  into  the  hands  of  the  warfighter. 
The  mission  is  to  provide  a  common  set  of  terms  and  conditions  [consistent  with  acquisition  policy  and 
doctrine  and  reconciled  with  Technology  Readiness  Levels  (TRLs)]  by  which  program  management 
teams  (at  all  levels)  can  assess  the  relative  maturity  and  risks  associated  with  technologies  being 
considered  to  meet  DoD  requirements.  The  goal  is  to  empower  all  members  of  the  acquisition  team 
with  pertinent  knowledge  of  the  risks— knowledge  that  is  needed  to  make  informed  decisions.  The 
objectives  are  to  provide  suggested  resources  (including  functional  concepts,  activities,  tools,  and 
resources)  available  for  program  risk  mitigation.  These  resources  include  most-promising  technologies, 
manufacturing  S&T  voids,  engineering  challenges,  industrial  shortfalls,  program  risks,  and  risk 
mitigation  tools  and  processes. 
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This  appendix  provides  an  overview  of  the  processes,  functional  communities, 
programs,  and  challenges  associated  with  identifying  and  mitigating  manufacturing- 
associated  risks  in  DoD  acquisition  programs.  Also,  where  possible,  it  lists  sources  that 
can  provide  information  about  strategies  or  approaches.  More  complete  guidance  on  DoD 
MRLs  will  be  forthcoming  in  a  DoD  MRL  Guidebook. 

1.2  MANDATORY/STATUTORY  REQUIREMENTS  AND  DoD  POLICY 
GUIDANCE 

DoD  Directive  (DoDD)  5000.1,  The  Defense  Acquisition  System,  dated  May  12, 
2003,  specifies  the  following: 

E.1.14.  Knowledge-Based  Acquisition.  PMs  shall  provide  knowledge 
about  key  aspects  of  a  system  at  key  points  in  the  acquisition  process.  PMs 
shall  reduce  technology  risk,  demonstrate  technologies  in  a  relevant  envi¬ 
ronment,  and  identify  technology  alternatives,  prior  to  program  initiation. 

They  shall  reduce  integration  risk  and  demonstrate  product  design  prior  to 
the  design  readiness  review.  They  shall  reduce  manufacturing  risk  and 
demonstrate  producibility  prior  to  full-rate  production. 

DoD  Instruction  (DoDI)  5000.2,  Operation  of  the  Defense  Acquisition  System, 
dated  May  12,  2003,  also  specifies  the  requirements  for  assessing  and  demonstrating  the 
manufacturing  readiness  of  a  system  at  various  stages  of  its  development.  Industrial  capa¬ 
bility  assessments  are  mandatory  requirements  at  Milestones  B  and  C.  In  addition  to 
mandatory/statutory  requirements,  DoDI  5000.2  provides  guidance  on  addressing  manu¬ 
facturing  and  production-related  risks  to  a  program.  These  sections  provide  the  acquisi¬ 
tion  manager  practical  guidelines  to  implement  the  laws  and  policies  relative  to  industrial 
capabilities.  They  also  provide  steps  a  manager  should  follow  to  integrate  defense  indus¬ 
trial  capabilities  considerations  into  the  acquisition  process  effectively  and  to  employ  the 
industry  in  acquisition  programs  effectively.  Table  1-1  provides  key  sections  on  produc¬ 
tion,  quality,  manufacturing,  and  industrial  capabilities-related  acquisition  policy  issues. 


Table  1-1.  Key  Sections  From  DoDI  5000.2 


Section 

Subject 

3.4 

User  Needs  and  Technology  Opportunities 

3.4.2 

Untitled  paragraph.  Discusses  technology  opportunities 

3.7 

System  Development  and  Demonstration 

3.7.1. 1 

Untitled  paragraph.  Discusses  the  purpose  of  the  SDD  phase 

3.7.4 

Proceeding  Beyond  the  Design  Readiness  Review 

3.7.5 

System  Demonstration 
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Table  1-1.  Key  Sections  From  DoDI  5000.2  (Continued) 


3.8 

Production  and  Deployment 

3.8.2 

Entrance  Criteria 

3.8.3 

LRIP 

3.8.4 

Full-Rate  Production  Criteria 

E3.  Enclosure  3 

Statutory,  Regulatory,  and  Contract  Reporting  Information  and  Milestone  Requirements 

Table  E3.T1 

Statutory  Information  Requirements 

E5.  Enclosure  5 

Integrated  Test  and  Evaluation 

E5.1.5 

Developmental  Test  and  Evaluation  (DT&E) 

E5.1.5.10 

Untitled  paragraph.  Discusses  how  to  demonstrate  the  maturity  of  the  production  process 

Defense  Federal  Acquisition  Regulations  Supplement  (DFARS)  207.105(b)(19) 
specifies  that,  as  part  of  the  acquisition  strategy,  program  managers  (PMs)  shall  perform 
an  analysis  of  the  capabilities  of  the  National  Technology  and  Industrial  Base  to  support 
the  design,  development,  sustained  production,  and  uninterrupted  maintenance  of  the 
system.  Specific  contents  of  the  industrial  capability  assessment  include 

•  The  availability  of  essential  raw  materials,  special  alloys,  composite  materi¬ 
als,  components,  tooling,  and  production  test  equipment  for  the  sustained 
production  of  systems  fully  capable  of  meeting  performance  objectives 
established  for  those  systems;  the  uninterrupted  maintenance  and  repair  of 
such  systems;  and  the  sustained  operation  of  such  systems 

•  Consideration  of  requirements  for  efficient  manufacture  during  the  design 
and  production  of  the  systems  to  be  procured  under  the  program 

•  The  use  of  advanced  manufacturing  technology,  processes,  and  systems 
during  the  research  and  development  (R&D)  phase  and  the  production  phase 
of  the  program. 

The  Defense  Acquisition  Guidebook  provides  policy  guidance  on  production, 
quality,  manufacturing,  and  industrial  capabilities  functional  topics  and  on  their  integra¬ 
tion  with  acquisition  critical  processes.  Section  2.3.16.1.4.5,  Industrial  Capability,  pro¬ 
vides  additional  guidance  on  elements  of  the  industrial  capability  assessments.  Table  1-2 
shows  other  key  sections. 


Table  1-2.  Key  Sections  From  the  Defense  Acquisition  Guidebook 


Chapter 

Subject 

2.3.7 

Systems  Engineering  Plan 

2.3.16.1.4 

Potential  Sources 

2.3.16.3 

Contract  Approach 
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Table  1-2.  Key  Sections  From  the  Defense  Acquisition  Guidebook  (Continued) 


Chapter 

Subject 

2.3.17 

Accounting  Review 

2.3.19 

Additional  Acquisition  Strategy  Topics 

3.1.4 

Implications  of  Evolutionary  Acquisition 

3.2.4 

Cost  As  An  Independent  Variable 

3. 7. 4. 2 

Assess  Risk  and  Sensitivity 

3.7.5 

System  Demonstration 

4.1.1 

Systems  Engineering 

4.1.5 

The  Integrated  Product  and  Process  Development  (IPPD)  Framework  and  Systems 
Engineering 

4. 2. 3. 2 

Technical  Planning 

4. 2. 3. 6 

Configuration  Management 

4. 2. 4. 4 

Implementation 

4.2.5. 1 

The  Use  of  Standards  versus  Capability  and  Maturity  Models 

4. 2. 5. 2 

Capability  Reviews 

4.3.3 

System  Development  and  Demonstration  Phase 

4. 3. 3. 4. 5 

Critical  Design  Review  (CDR) 

4. 3. 3. 5 

Outputs  of  the  Systems  Engineering  Processes/Inputs  to  the  Design  Readiness  Review 

4. 3. 3. 6 

Purpose  of  Systems  Engineering  in  System  Demonstration 

4. 3. 3. 8. 4 

Combined  Developmental  Test  and  Evaluation,  Operational  Test  and  Evaluation,  and 

Live  Fire  Test  and  Evaluation  Demonstrate  System  To  Specified  User  Needs  and  Envi¬ 
ronmental  Constraints 

4. 3. 3. 9. 2 

System  Verification  Review  (SVR) 

4. 3. 3. 9. 3 

Production  Readiness  Review  (PRR) 

4.3.4. 1 

Purpose  of  Systems  Engineering  in  Production  and  Deployment 

4. 3. 4. 4. 3 

Physical  Configuration  Audit  (PCA) 

4.3.5. 1 

Purpose  of  Systems  Engineering  in  Operations  and  Support 

4.4.4 

Software 

4.4.6 

Manufacturing  Capability 

4. 4. 6. 2 

Manufacturing  Readiness  Levels 

4.4.7 

Quality 

4.4.8 

Reliability,  Availability,  and  Maintainability  (RAM) 

4.5.6 

Trade  Studies 

4. 5. 7. 3 

Modeling  and  Simulation  (M&S)  in  Systems  Development  and  Demonstration 

4. 5. 7. 4 

Modeling  and  Simulation  (M&S)  in  Production  and  Development 

5.2.1. 5 

Continuous  Technology  Refreshment  and  Obsolescence 

5.2.2 

Pre-Acquisition  and  Acquisition  (Design  for  Support) 
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Table  1-2.  Key  Sections  From  the  Defense  Acquisition  Guidebook  (Continued) 


Chapter 

Subject 

5.4.1. 1.2 

Life-Cycle  Logistics  (LCL)  Considerations  During  Concept  Refinement 

5.4.2. 1 

System  Development  and  Demonstration  Leading  to  Milestone  C 

6.4.1 

Integrated  Product  and  Process  Development  and  Integrated  Product  Teams 

6. 4. 5. 2 

Technology  Development  and  System  Development  and  Demonstration 

7. 8. 3. 3 

Redesigning  the  Processes  That  the  Acquisition  Supports 

8.4.4. 1 

Risk  Management  in  Systems  Engineering 

8.4.5. 1 

Critical  Program  Information  (CPI) 

9.1. 3.3 

Capability  Production  Document  (CPD) 

9.3 

Developmental  Test  and  Evaluation 

9.10 

Test  and  Evaluation  Master  Plan  Recommended  Format 

11.2.1.1 

International  Considerations  and  Program  Strategy 

11.3.5 

Quality 

11.5 

Knowledge-Based  Acquisition 

11.8 

Integrated  Product  and  Process  Development  (IPPD) 

11.13 

Simulation-Based  Acquisition  (SBA)  and  Modeling  and  Simulation  (M&S) 

Guidance  on  the  transition  from  development  to  production  can  be  found  at  the 
AT&L  Knowledge  Sharing  System  (AKSS)  Web  site.  AKSS,  which  is  part  of  the 
Defense  Acquisition  University  Knowledge  System,94  was  launched  in  October  2002  to 
replace  the  Defense  Acquisition  Deskbook  (DAD).  Like  its  predecessor,  AKSS  continues 
to  provide  acquisition  information  for  all  DoD  Service  components  and  across  all  func¬ 
tional  disciplines.  It  emphasizes  the  need  for  a  rigorous,  disciplined  application  of  fun¬ 
damental  engineering  principles,  methods,  and  techniques  and  the  identification  and 
assessment  of  program  risk  elements  throughout  the  acquisition  cycle.  Finally,  it  provides 
“templates”  designed  to  introduce  discipline  into  the  acquisition  process,  to  identify  and 
give  visibility  to  high-risk  factors,  and  to  provide  the  tools  by  which  risk  can  be  mini¬ 
mized  progressively  in  major  risk  categories  (funding,  design,  test,  production,  transition 
plan,  facilities,  logistics,  and  management). 


94  AKSS  is  one  part  of  the  Defense  Acquisition  University  Knowledge  System.  The  two  sites  are  the 
Acquisition  Knowledge  Sharing  System  (http://www.deskbook.osd.mil/jsp/default.jsp)  and  the 
Acquisition  Community  Connection  (http://www. deskbook. osd.mil/jsp/AkssPage.jsp?fName=../jsp/ 
community  _central.jsp&title=AKSS%20Community%20Central). 
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1.3  KEY  ELEMENTS  (THREADS)  OF  THE  INDUSTRIAL  PROCESS 


Understanding  the  risks  associated  with  the  industrial  process  in  DoD  acquisition 
and  developing  risk  mitigation  plans  and  action  are  central  to  successful  acquisition  pro¬ 
gram  management.  These  risk  elements  are  discrete  (for  each  phase)  and  constitute  nine 
threads  that  transcend  the  transition  from  discovery  and  invention,  through  engineering 
and  development,  to  production  and  deployment,  and  eventual  disposal. 

1.3.1  Technology  and  Industrial  Base  Thread 

This  thread  requires  an  analysis  of  the  capabilities  of  the  national  technology  and 
industrial  base  to  support  the  design,  development,  production,  operation,  uninterrupted 
maintenance  support  of  the  system,  and  eventual  (environmentally  conscious)  disposal. 

Key  issues  for  the  Technology  and  Industrial  Base  Thread  include 

1.  Technology  base  maturity  [Technology  Readiness  Level  (TRLs)] 

2.  Technology  leadership  (domestic  vs.  foreign  and  commercial  vs.  Govern¬ 
ment) 

3.  Manufacturing  technology  voids 

4.  Industrial  sector  structure,  trends,  capabilities,  and  capacities  (including 
potential  subcontractors,  suppliers,  and  vendors). 


1.3.2  Design  Thread 

This  thread  requires  an  analysis  of  the  degree  to  which  the  identified/evolving 
system  design  will  meet  user  requirements  and  the  degree  to  which  the  design  is  new  and 
unproven. 

Key  issues  for  the  Design  Thread  include 

1.  Design  approach,  maturity  (percentage  of  design  that  is  new),  and  stability 

2.  Design  analyses  and  tools,  Design  for  Excellence  (DFX)  (X  =  producibility 
engineering,  design  for  manufacturing  and  assembly,  testability,  cost  effec¬ 
tiveness,  and  other  planning  efforts) 

3.  Use  of  multifunctional  Integrated  Process  Teams  (IPTs)  (includes  manu¬ 
facturing  considerations  as  tradeoffs) 

4.  Configuration  and  block  change  management 

5.  Manufacturing,  testability,  and  methods  improvement 

6.  Manufacturing  management  issues  in  design  reviews  and  manufacturing- 
specific  reviews  (including  Manufacturing  Feasibility  Reviews, 
Manufacturing  Capability  Risk  Reviews,  Producibility  Trade  Studies  and 
Reviews,  and  Production  Readiness  Reviews). 
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1.3.3  Materials  Thread 


This  thread  requires  and  analysis  of  the  risks  associated  with  materials  (including 
basic/raw  materials,  components,  semi-finished,  parts,  and  subassemblies). 

Key  issues  for  the  Materials  Thread  include 

1.  Availability  and  degree  of  competition 

2.  Sources  (domestic/foreign/single/sole/diminishing)  and  Make/Buy  Plan 

3.  Use  of  commercial-of-the-shelf/non-developmental  items/commercial  items 

4.  Costs,  lead  times,  and  capacity  constraints  and  scale-up  challenges 

5.  Understanding  materials’  basic  properties  and  environmental  considerations 

6.  Characterization  in  a  manufacturing  environment 

7.  Storage,  handling,  and  parts  control. 


1.3.4  Cost  and  Funding  Thread 

This  thread  requires  an  analysis  of  the  risk  that  the  system  development  and 
deployment  will  not  meet  the  DoD  cost  and  funding  goals. 

Key  issues  for  the  Cost  and  Funding  Thread  include 

1.  Early  manufacturing  involvement  in  technology  development  and  selection 

2.  Establishment  of  design-to-cost  (DTC)  and  manufacturing  cost  goals 

3.  Cost-reduction  activities 

4.  Progress  toward  meeting  goals 

5.  Availability  of  necessary  funding 

6.  Plans  for  cost  mitigation. 


1.3.5  Process  Capability  and  Control  Thread 

This  thread  requires  an  analysis  of  the  risk  that  the  manufacturing  processes  may 
not  be  able  to  reflect  (repeatably  and  affordably)  in  the  design  of  key  characteristics. 

Key  issues  for  the  Process  Capability  and  Control  Thread  include 

1.  Process  characterization 

2.  Variation  and  variability  reduction 

3.  Identification  of  key  characteristics  and  process  capability  indexes 

4.  Sigma  levels. 
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1.3.6  Quality  Management  (QM)  Thread 


This  thread  requires  an  analysis  of  the  risk  and  management  efforts  to  control 
quality  and  foster  continuous  quality  improvement. 

Key  issues  for  the  QM  Thread  include 

1.  Planning  for  quality 

2.  The  quality  organization  and  strategy 

3.  Prime  contractor  QM  plan 

4.  Key  supply  chain  QM  structures 

5.  Understanding  the  contractor’s  quality  model 

6.  Deployment  of  risks  into  contract  language 

7.  Coordination  with  Defense  Contract  Management  Agency  (DCMA) 
resources. 


1.3.7  Personnel  Thread 

This  thread  requires  the  assessment  of  the  skills  and  availability  of  the  people 
required  to  support  the  manufacturing  effort. 

Key  issues  for  the  Personnel  Thread  include 

1.  Involvement  with  the  S&T  and  Manufacturing  Technology  programs 

2.  Manufacturing  involvement  in  the  systems  engineering  and  IPPD  processes 

3.  Manufacturing  planners,  schedulers,  and  control  personnel 

4.  Tooling  and  industrial  engineers 

5.  Process  operators  (including  training  plans  and  required  certifications). 


1.3.8  Facilities  Thread 

This  thread  requires  an  analysis  of  the  capabilities  and  capacity  (prime,  subcon¬ 
tractor,  supplier,  vendor,  maintenance,  and  repair)  that  are  key  risks  in  manufacturing. 

Key  issues  for  the  Facilities  Thread  include 

1.  Location  (domestic  or  foreign) 

2.  New  or  existing  lines 

3.  Dedicated  or  shared 

4.  Commercial  or  traditionally  defense 

5.  Government  or  contractor  owned/operated  (organic,  commercial,  or  core) 

6.  Local  environmental  laws  and  regulations 

7.  Labor  unions 

8.  Capacity  utilization 

9.  Use  of  manufacturing  development  centers/pilot  lines. 


1.3.9  Manufacturing  Planning,  Scheduling,  and  Control  Thread 


This  thread  requires  an  analysis  of  the  integration  of  all  the  elements  needed  to 
translate  the  design  into  an  integrated  and  fielded  system  (meeting  program  goals  for 
affordability  and  availability). 


Key 

include 

issues  for  the  Manufacturing  Planning,  Scheduling,  and  Control  Thread 

1. 

Adequacy  of  the  manufacturing  strategy 

2. 

Integration  with  the  acquisition  strategy 

3. 

Maturity  of  the  manufacturing  plan 

4. 

Integration  with  the  risk  management  plan 

5. 

Scheduling  tooling 

6. 

Capital  equipment  installation  and  maintenance 

7. 

Personnel 

8. 

Deliveries  (i.e.,  materiel  management) 

9. 

Product  flow  and  test  equipment 

10. 

Supply  chain  management. 

In  support  of  these  threads,  the  AT&L  production,  manufacturing,  and  quality 
function  has  promulgated  a  set  of  core  activities  designed  to  address  associated  risks  and 
mitigation  activities  (see  Figure  1-1).  Within  each  acquisition  phase,  a  mature  body  of 
knowledge,  doctrine,  and  tools  for  critical  processes  supports  these  activities. 

1.4  MRLs 

MRLs  are  designed  to  provide  a  standard  set  of  functional  definitions  and  terms 
for  the  AT&L  community  so  that  it  can  accomplish  its  tasks  and  reporting  requirements. 
These  MRLs  provide  a  common  language  for  the  S&T  and  acquisition  communities  in 
the  following  areas: 

•  DoD  investments  in  Basic  Research  [i.e.,  Program  Element  (PE)  6.1)].  An 

early  understanding  of  the  basic  principles  observed  and  reported  could 
benefit  from  knowledge  of  ongoing  research  activities  into  the  understanding 
of  basic  manufacturing  sciences  and  theories. 

•  DoD  investments  into  Exploratory  Research  (i.e.,  PE  6.2).  DoD  S&T  PMs 

would  benefit  from  a  knowledge  of  advanced  manufacturing  technologies 
(including  materials,  processes,  and  systems).  This  knowledge  will  assist 
them  in  making  informed  decisions  on  selections  of  the  most  promising  tech¬ 
nologies  and  materials  being  considered  for  DoD  (from  an  eventual  manufac¬ 
turing  cost  and  schedule  baseline  perspective  for  early  tradeoffs). 
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Concept 

Technology 

System  Development 

Production  &  Operations  & 

Refinement 

Development 

&  Demonstration 

Deployment  Support 

Design 

FRP 

\  Concept 

/\  Readiness 

LRIP  /\  Decision 

/Decision  A  A  V  Review  /\IOT&E  V  Review 

Manufacturing  Threads: 


Ind.  Base 


Design 


Key  Manufacturing  Tasks 


Industrial  Base  Issues 


•  Dual  Use  vs  Spin-On/Off 

•  Materials  Science *  *  Subcontractor  Selection  (Make/Buy) 

•  Technology  Voids  *  Vendor/Supplier  Selection 

•  Foreign  Component  Technology  Dependencies 

•  Manpower  &  Skills  Availability 


•  Producibility  Engineering  & 

Planning 

•  Design  Approach 


Influence  the  Design 

•  Producibility  .  Productioi 
Assessments 

•  Design  Analysis  &  tools  (DFX) 


n  Readiness  Reviews  •  Value  Engineering 
Design  Stability 


Materials 


Cost/Funding 


•  Availability 


Materials  Science  and  Materials  Management 

•  Material  Process  .  Material  Process 
Capabilities  Proofing 

•  Supply  Chain  Mgmt. 


Scale-up 
Lead  Times 


-  Evaluate  Manufacturing  Costs 

•  Identity  Mfg.  Cost  Drivers  •  Evaluate  Mfg.  Cost  Drivers 

•  Funding  Availability  •  Develop  Mitigation  Plans 


•  Second  Source 
Decisions 


•  Track  Mfg.  Cost  Drivers 

•  Identify  Cost  Reduction  Activities 


Process  Capability/-* - j -  Process  Capability  and  Control  - 1 - 1 

Control  •  Process  characterization  •  Key  Characteristics  •  Process  Capability  •  Variability  Reduction  •  Six  Sigma 


Quality  Mgmt. 


Personnel 


•  Initial  QA  Planning 


Quality  Assurance  Planning 

•Organizing  •  Supply  Chain  QA 


Personnel 
Skilled  &  Available 


IQUE  Process 


•  Program  &  DCMA  Oversight 

•  Metrology  &  Calibration 


-  Involved  with  S&T  programs  •  Tooling  Engineers 

•  Involved  with  ManTech  programs  *  Manufacturing  Engineering  .  industrial  Engineers 
•  Manufacturing  Planning 


•  Process  Engineers 

*  Environmental  Engineers 


•  Facilities 


•  Foreign  or  Domestic 

•  New  or  Existing  Lines 


Facilities 

Capability  and  Capacity 

•  Commercial  or  DoD  •  Dedicated  or  •  Env.  Laws 

Shared  Lines  •  Development  Ctr. 


•  Utilization 

•  Unions 


•  Mfg.  Planning  _  Preliminary  _  Final  _  Executive  _ 

Manufacturing  Plan  Manufacturing  Plan  Manufacturing  Plan 

•  Acq.  Strategy  •  Mature  Mfg.  Plan  •  Schedule  tooling  •  Test  equipment 

•  Mfg.  Strategy  •  Integrate  with  Risk  Mgmt.  •  Work  Measurement  •  Inspection  equipment 

•  Capital  Equipment  •  Type  II  Standards  •  Type  I  Standards 


Figure  1-1.  Associated  Risks  and  Mitigation  Activities 
[Source:  Defense  Acquisition  University  (DAU)  Course  Materials] 

•  Upon  approval  for  entry  into,  for  example,  Advanced  Technology  Dem¬ 
onstration  (ATD),  Advanced  Concept  Technology  Demonstration 
(ACTD),  or  Milestone  B.  DoD  R&D  activities  must  address  the  industrial 
and  manufacturing  capabilities  (and  risks)  needed  to  support  programs.  Tech¬ 
nical  issues  include  design  performance  (including  reliability  and  product 
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variation).  Business  issues  include  cost  (i.e.,  design,  producibility,  contractor 
methods,  and  improvement  incentives),  schedule  (i.e.,  facility  manufacturing 
capabilities  and  capacities),  and  performance  (i.e.,  contractor  quality  and 
environmental  safety  and  health  considerations). 

•  Sustaining  the  operational  system.  This  includes  a  knowledge  of  industrial 
technologies,  capabilities,  capacities,  and  (eventual)  disposal  risks. 


1.4.1  MRL  Levels,  Definitions,  Descriptions,  and  Acquisition  Phases 

Table  1-3  shows  the  current  iteration  of  proposed  DoD  MRLs.  The  MRL  defini¬ 
tions  and  descriptions  are  based  on  the  integration  of  existing  industry,  government,  and 
technical  coalition  standards  and  recommendations.  MRLs  1  to  3  have  been  added  to  tie 
ongoing  DoD  investments  and  activities  in  manufacturing  science  and  advanced  manu¬ 
facturing  technology  explorations  to  corresponding  TRLs.  MRL  10  has  been  added  to 
emphasize  continuous  process  improvement  in  the  industrial  environment.  MRL  defini¬ 
tions  have  been  refined  to  incorporate  DoD  S&T  and  acquisition  doctrinal  standard  phra¬ 
seology.  MRL  descriptions  have  been  organized  to  provide  a  comprehensive  set  of  core 
risk  categories  (threads)  that  transcend  all  levels  of  manufacturing  readiness. 


Table  1-3.  Current  Iteration  of  Proposed  DoD  MRLs 


MRL 

Definition 

Description 

Acquisition 

Phase 

1-3 

Manufacturing  concepts 
Identified. 

Identification  of  current  manufacturing  concepts  or  produci¬ 
bility  needs  based  on  laboratory  studies. 

Pre-Concept 

Refinement. 

4 

System,  component,  or 
item  validation  in  a  labo¬ 
ratory  environment. 

This  is  the  lowest  level  of  production  readiness.  Technolo¬ 
gies  must  have  matured  to  at  least  TRL  4.  At  this  point,  few 
requirements  have  been  validated,  and  there  are  large  num¬ 
bers  of  engineering/design  changes.  Component  physical 
and  functional  interfaces  have  not  been  defined.  Materials, 
machines,  and  tooling  have  been  demonstrated  in  a  labo¬ 
ratory  environment.  Inspection  and  test  equipment  have 
been  demonstrated  in  a  laboratory  environment.  Manufac¬ 
turing  cost  drivers  are  identified.  Producibility  assessments 
have  been  initiated. 

Concept 
Refinement 
leading  to  a 
Milestone  A 
decision. 

5 

System,  component,  or 
item  validation  in  initial 
relevant  environment. 
Engineering  application/ 
breadboard,  brassboard 
development. 

Technologies  must  have  matured  to  at  least  TRL  5.  At  this 
point,  all  requirements  have  not  been  validated,  and  there 
are  significant  engineering/design  changes.  Component 
physical  and  functional  interfaces  have  not  been  defined. 
Materials,  machines,  and  tooling  have  been  demonstrated  in 
a  relevant  manufacturing  environment,  but  most  manufac¬ 
turing  processes  and  procedures  are  in  development  (or 
ManTech  initiatives  are  ongoing).  Inspection  and  test  equip¬ 
ment  have  been  demonstrated  in  a  laboratory  environment. 
Production  cost  drivers/goals  are  analyzed.  System-level 

DTC  goals  are  set.  Producibility  assessments  ongoing. 

Technology 
Development 
leading  to  a 
Milestone  B 
decision. 
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Table  1-3.  Current  Iteration  of  Proposed  DoD  MRLs  (Continued) 


MRL 

Definition 

Description 

Acquisition 

Phase 

6 

System,  component  or 
item  in  prototype  demon¬ 
stration  beyond  bread¬ 
board,  brassboard 
development. 

During  the  prototype  demonstration  phase,  requirements  are 
validated  and  defined.  However,  there  are  still  many  engi¬ 
neering/design  changes,  and  physical  and  functional  inter¬ 
faces  are  not  yet  fully  defined.  Technologies  must  have 
matured  to  at  least  TRL  6.  Raw  materials  are  initially  dem¬ 
onstrated  in  relevant  manufacturing  environment.  Similar 
processes  and  procedures  have  been  demonstrated  in 
relevant  manufacturing  environment.  At  this  point,  there  are 
likely  major  investments  required  for  machines  and  tooling. 
Inspection  and  test  equipment  should  be  under  develop¬ 
ment.  Producibility  risk  assessments  ongoing  and  trade 
studies  conducted.  A  production  Cost  Reduction  Plan  is 
developed.  Production  goals  are  met. 

System  Devel¬ 
opment  and 
Demonstration 
(SDD)  leading 
to  Design 
Readiness 

Review  (DRR). 

7 

System,  component  or 
item  in  advanced 
development. 

Technologies  must  have  matured  to  at  least  TRL  7.  At  this 
point,  engineering/design  changes  should  decrease.  Physi¬ 
cal  and  functional  interfaces  should  be  clearly  defined.  All 
raw  materials  are  in  production  and  available  to  meet 
planned  Low  Rate  Initial  Production  (LRIP)  schedule.  Pilot 
line  manufacturing  processes  and  procedures  set  up  and 
under  test.  Processes  and  procedures  not  yet  proven  or 
under  control.  During  this  phase,  initial  producibility  improve¬ 
ments  should  be  underway.  DTC  estimates  are  less  than 

125  percent  of  goals.  Detailed  production  estimates  are 
established. 

SDD;  post  DRR. 

8 

System,  component  or 
item  in  advanced  devel¬ 
opment.  Ready  for  LRIP. 

Technologies  must  have  matured  to  at  least  TRL  8.  At  this 
point,  engineering/design  changes  should  decrease  signifi¬ 
cantly.  Physical  and  functional  interfaces  should  be  clearly 
defined.  All  raw  materials  are  in  production  and  available  to 
meet  planned  LRIP  schedule.  Manufacturing  processes  and 
procedures  have  been  proven  on  the  pilot  line  and  are  under 
control  and  ready  for  LRIP.  During  this  phase,  initial  pro¬ 
ducibility  risk  assessments  should  be  completed.  Production 
cost  estimates  meet  DTC  goals. 

SDD  leading  to 
a  Milestone  C 
decision. 

9 

System,  component,  or 
item  previously  produced 
or  in  production, 
or 

the  system,  component, 
or  item  is  in  LRIP.  Ready 
for  Full  Rate  Production 
(FRP). 

During  LRIP,  all  systems  engineering/design  requirements 
should  be  met,  and  there  should  only  be  minimal  system 
engineering/design  changes.  Technologies  must  have 
matured  to  at  least  TRL  9.  Materials  are  in  production  and 
available  to  meet  planned  production  schedules.  Manufac¬ 
turing  processes  and  procedures  are  established  and 
controlled  in  production  to  three-sigma  or  some  other  appro¬ 
priate  quality  level.  Machines,  tooling,  and  inspection  and 
test  equipment  deliver  three-sigma  or  some  other  appro¬ 
priate  quality  level  in  production.  Production  risk  monitoring 
is  ongoing.  LRIP  actual  costs  meet  estimates. 

Production  and 
deployment 
leading  to  an 

FRP  decision. 
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Table  1-3.  Current  Iteration  of  Proposed  DoD  MRLs  (Continued) 


MRL 

Definition 

Description 

Acquisition 

Phase 

10 

System,  component,  or 
item  previously  produced 
or  in  production, 
or 

the  system,  component, 
or  item  is  in  FRP. 

The  highest  level  of  production  readiness.  Minimal  engin¬ 
eering/design  changes.  System,  component,  or  item  is  in 
production  or  has  been  produced  and  meets  all  engineering, 
performance,  quality,  and  reliability  requirements.  All  mate¬ 
rials,  manufacturing  processes  and  procedures,  and  inspec¬ 
tion  and  test  equipment  are  controlled  in  production  to  six- 
sigma  or  some  other  appropriate  quality  level  in  production. 

A  proven,  affordable  product  is  able  to  meet  the  required 
schedule.  Production  actual  costs  meet  estimates. 

FRP/ 

sustainment 

MRLs  present  a  way  to  operationally  define,  measure,  and  manage  manufacturing 
maturity  (i.e.,  through  a  structured  approach  and  using  proven  tools  and  techniques  to 
identify  and  quantify).  Once  identified,  these  MRLs  provide  visibility  into  manufacturing 
and  program  risk  areas  and  can  be  used  as  a  part  of  a  comprehensive  risk  identification 
and  mitigation  program.  As  such,  they  must  address  the  basic  elements  of  a  manufac¬ 
turing  process. 

When  DoD  teams  address  MRL  accomplishments  at  each  acquisition  Milestone 
decision,  they  need  a  common  set  of  manufacturing  risk  categories  for  each  phase  of 
development.  These  categories  should  build  from  lower  to  higher  levels,  be  tied  to  the 
acquisition  program’s  knowledge  maturation,  and  incorporate  standard  phraseology  and 
terms  and  approved  metrics. 

MRL  content,  focus,  and  depth  of  knowledge  change  as  a  technology  proceeds 
from  discovery,  through  system  development,  onto  the  factory  floor,  and  into  the  hands 
of  the  warfighter.  Some  MRLs  (i.e.,  1-4)  are  Pre-Milestone  A.  Other  MRLs  (i.e.,  5-8)  are 
designed  to  support  Milestones  B  and  C  and  also  LRIP  and  FRP  decisions.  MRL  9  is 
designed  to  support  risk  management  needs  for  DoD  production  and  reprocurement  (i.e., 
spares  and  maintenance)  activities.  MRL  10  is  designed  to  support  FRP. 

1.4.2  MRL  Exit  Criteria 

Table  1-4  provides  suggested  exit  criteria  and  ties  to  TRLs  and  acquisition  phase 
requirements  against  manufacturing  doctrinal  key  elements,  core  activities,  and  critical 
processes.  The  ultimate  goal  is  to  provide  knowledge  of  manufacturing  risks  to  a  given 
program  (at  various  stages  of  development)  to  enable  decision-makers  (at  all  levels)  to 
make  informed  decisions  on  risk  mitigation  strategies  and  plans.  MRLs  must  be  defined 
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for  each  technology.  In  addition,  they  must  progress  within  the  context  of  a  system-level 
(i.e.,  material,  piece  part,  component,  subassembly,  and  system)  perspective.  They  should 
be  an  integral  part  of  a  program’s  acquisition  and  risk  management  plans  and  strategies. 
Finally,  MRLs  should  address  program  risks  using  standard  information  obtained  as  part 
of  the  IPPD  process  by  IPTs. 

1.4.3  MRL  Risk  Roll-Up  Assessments  for  Systems  and  Programs 

The  DoD  MRL  process  is  envisioned  as  a  two-phased  effort.  The  first  phase  consists  of  a 
discrete  (i.e.,  bottoms-up)  assessment  of  the  relative  maturity  of  a  technology  (from  a 
manufacturing  and  quality  perspective)  against  DoD  program  goals  and  objectives.  The 
second  phase  is  designed  to  compile  the  findings  of  individual  MRL  assessments  into  a 
concise  format  for  analysis  and  decision-making  at  subsequent  levels  of  the  program, 
including  program  risks  and  recommended  risk  mitigation  actions. 

At  each  stage  of  development,  a  technology  should  be  assessed  and  evaluated  for 
relative  risk  in  meeting  established  program  goals.  Each  MRL  should  be  identified  at  the 
appropriate  risk  level  (i.e.,  low,  medium,  or  high).  Figure  1-2  provides  a  format. 


Figure  1-2.  Metric  Roll-up  Chart 

[Source:  MDA  Engineering  and  Manufacturing  Readiness  Level  (EMRL) 
Draft  Implementation  Guide ] 


Note  for  Figure  1-2: 

Green:  Complete. 

Not  complete;  no  effect  on  cost  or  schedule. 

Red:  Not  complete;  significant  effect  on  cost  or  schedule. 
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During  the  second  phase  of  MRL  assessments,  program  management  should 
assess  the  risks  of  lower-tier  MRLs  (i.e.,  components  and  parts)  for  subsequent  (i.e., 
higher-order)  risk  mitigation  efforts  (see  Figure  1-3).  No  magic  formula  exists  for  rolling 
up  the  effects  of  components,  assemblies,  or  subsystems  into  one  system-level  metric. 
One  of  the  options  that  make  the  most  sense  is  to  establish  weighted  guidelines  that  take 
into  account  the  criticality  of  an  emerging  high-risk  technology.  It  is  critically  important 
to  understand  that  a  single  high-risk  technology  could  be  a  program  showstopper. 


EMRL  STATUS  REVIEW 
(SEEKER) 

Problem:  The  baseline  Focal  Plane  Array  (FPA)  is  experiencing  low  yields  and 
high  failure  rates  during  testing. 

Solution  1: 

•  MANTECFI  program  to  improve  yields  and  lower  failure  rates: 

•  Cost  of  program  $600K 

•  Schedule  Slip  7  Months  from  receipt  of  funding 

•  Risk  Low 

Solution  2:  (Suggested) 

•  Company  (B)  produces  FPA  and  Readout  that  can  be  used  in  place  of  current 
FPA,  but  FPA  may  slightly  degrade  seeker  performance.  Performance  impact 
assessed  as  moderate. 

•  Cost  for  change  $800K 

•  Schedule  Slip  None  (if  under  control  within  60  -  90  days) 

•  Risk  Low 


Figure  1-3.  Sample  EMRL  Status  Review  (Seeker) 

(Source:  MDA  EMRL  Draft  Implementation  Guide) 

The  status  of  MRLs  that  do  not  meet  phase  goals  should  be  reviewed,  and  pro¬ 
gram  management  must  make  a  decision  among  alternatives  (see  Figure  1-3): 

•  System  element  (i.e.,  component) 

•  Problem  (i.e.,  baseline  component  yields  and  failure  rates  during  testing) 

•  Program  effects  (i.e.,  cost,  schedule,  and  technical  risks) 

•  Alternative  solutions  (i.e.,  technical,  cost,  schedule,  and/or  business). 
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1.4.4  Acquisition  Risk  Mitigation  Actions 

As  with  all  acquisition  program  risks,  those  associated  with  industrial  capabilities, 
manufacturing,  production,  and  quality  will  require  an  integrated  risk  management  strat¬ 
egy  and  plans  for  the  program.  Risk  handling  strategies  and  mitigation  efforts  will 
address  two  areas:  technical  and  business.  MRLs  provide  the  IPTs  with  a  focus  on  major 
program  functional  risks,  a  doctrinal  body  of  knowledge  of  key  risk  areas,  and  core  com¬ 
petencies  for  risk  identification  and  mitigation  activities. 

Additional  information  is  provided  on  suggested  manufacturing  management  risk 
identification  and  management  activities,  tools,  and  techniques  (see  Subsection  1.5).  Also 
provided  is  a  list  of  manufacturing  management-related  Web  sites  as  a  resource  for 
obtaining  detailed  information  on  this  body  of  knowledge  (see  Subsection  1.6). 

1.5  RECOMMENDED  TOOLS  FOR  MANUFACTURING  RISK  IDENTIFI¬ 
CATION  AND  MITIGATION 

Note:  These  are  only  some  of  the  tools  that  are  available  to  manufacturing  man¬ 
agers.  The  inclusion  of  these  tools  in  this  Deskbook  does  not  constitute  an  endorsement  of 
any  individual  or  company.  The  tools  are  listed  in  alphabetical  order. 

1.5.1  Advanced  Quality  Systems  (AQSs) 

AQSs  are  designed  to  provide  suppliers  the  tools  and  the  training  needed  so  that 
the  supplier  or  vendor  can  deliver  quality  products  and  services  on  time.  AQSs  involve 
the  systematic  reduction  of  variation  in  key  characteristics.  AQS  process  steps  include 

•  Identifying  key  characteristics. 

•  Determining  where  key  characteristics  will  be  measured  and  setting  up  con¬ 
trol  charts. 

•  Collecting  data  and  determining  if  the  key  characteristic  is  “in  control.”  If  not 
in  control,  determining  the  source  of  variation  and  remove  the  special  causes 
of  variation. 

•  Establishing  controls  for  key  process  parameters  and  document  in  the  AQS 
control  plan. 

1.5.2  Design  for  Manufacturing  and  Assembly  (DFMA)™ 

DFMA™  is  a  systematic  analysis  of  an  assembly  to  simplify  its  design,  assembly, 
and  manufacturing  capabilities  without  effecting  performance.  The  analysis  supports  the 
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determination  of  the  theoretical  minimum  number  of  parts  that  must  be  in  the  design  for 
the  product  to  function  as  required.  Manufacturing  costs  are  reduced  as  unnecessary  parts 
are  identified  and  eliminated. 

1.5.3  Design  of  Experiments  (DOE) 

DOE  is  a  structured,  organized  method  for  determining  the  relationship  between 
factors  and  the  output  of  a  process.  This  knowledge  allows  engineers  to  optimize  the 
design  (robust)  and  improve  quality,  reliability,  and  performance  while  reducing  costs. 
The  goal  is  to  achieve  a  “robust  design”  (i.e.,  a  design  that  performs  as  intended  regard¬ 
less  of  variation  in  a  product’s  manufacturing  process).  Off-line  quality  is  a  tool  used  by 
design  and  production  engineering  to  maximize  functional  quality  by  designing  and 
building  quality  in. 

1.5.4  International  Organization  for  Standardization  (ISO)  9001:2000 

ISO  9001:2000  is  a  series  of  international  quality  standards  used  by  companies 
and  organizations  to  provide  a  basis  for  ensuring  customers  that  processes  and  procedures 
that  will  help  to  ensure  quality  products  and  services  are  in  place.  The  intent  of  the  ISO 
9001:2000  series  is  to  provide  consistent  customer  satisfaction  and  improved  competi¬ 
tiveness.  ISO  9001:2000  quality  standard  consists  of  five  sections;  Quality  Management 
System,  Management  Responsibility,  Resource  Management,  Product  Realization  and 
Measurement,  Analysis  and  Improvement. 

1.5.5  Lean-Pathways® 

Lean -Pathways®  applies  lean  manufacturing  practices  on  key  suppliers  based  on 
their  strategic  importance  to  the  program.  Suppliers  complete  a  series  of  diagnostic  tools, 
training,  data  collection,  and  process  mapping  exercises  designed  to  teach  them  how  to 
identify  improvement  opportunities.  The  teams  identify  and  implement  solutions  during 
Accelerated  Improvement  Workshops  (Kaizen  events)  to  reduce  cycle  times  and  improve 
work  processes.  The  Lean-Pathways  process  includes  conducting  internal  operations 
review  and  mapping  value  streams,  developing  a  future  state  analysis  and  identifying 
gaps  and  setting  priorities,  conducting  in-plant  opportunity  reviews,  developing  a  road¬ 
map,  hosting  improvement  workshops,  and  implementing  solutions. 
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1.5.6  Malcolm  Baldrige  National  Quality  Award  Criteria95 

The  Malcolm  Baldrige  National  Quality  Award  criteria  are  used  to  improve  per¬ 
formance  and  competitiveness  and  ensure  delivery  of  value  to  the  customer.  The  award 
promotes  quality  awareness  and  publicizes  how  businesses  and  organizations  can  use 
quality  improvement  programs  to  improve  performance  and  achieve  excellence.  Baldrige 
criteria  have  been  developed  for  seven  key  business  areas:  (1)  leadership,  (2)  strategic 
planning,  (3)  customer  and  market  focus,  (4)  measurement,  analysis,  and  knowledge 
management,  (5)  human  resource  focus,  (6)  process  management,  and  (7)  business 
results.  Improvement  is  based  on  the  establishment  of  a  documented  approach,  the 
deployment  of  that  approach,  and  the  results  attained. 

1.5.7  Manufacturing  Technology  (ManTech)  Programs 

DoD,  Army,  Navy,  Air  Force,  and  DLA  ManTech  programs  contribute  to  the 
ability  of  the  U.S.  industrial  base  to  develop,  mature,  and  produce  military  equipment  that 
is  affordable  and  presents  a  low-risk  production  environment.  The  ManTech  program 
addresses  production  issues  early— from  system  development  through  transition  to  pro¬ 
duction  and  sustainment— by  identifying  and  funding  manufacturing  risk  areas  and  tech¬ 
nologies.  The  JDMTP  is  organized  to  identify  and  integrate  requirements,  conduct  Joint 
program  planning,  develop  Joint  strategies,  and  oversee  the  execution  of  ManTech 
programs. 

1.5.8  Quality  Function  Deployment  (QFD) 

QFD  is  a  structured  methodology  used  to  capture  requirements  (the  voice  of  the 
customer),  which  then  drive  the  design.  QFD  is  a  matrix  that  provides  visibility  into  the 
customer  requirements  and  design  process.  This  matrix  gives  the  engineers  a  structure  for 
examining  all  of  the  requirements  to  ensure  that  they  have  developed  design  solutions 
that  will  meet  those  needs.  The  matrix  is  made  up  of  what’s,  how’s,  customer  percep¬ 
tions,  and  other  factors  that  will  eventually  be  used  to  drive  design  solutions  and  ulti¬ 
mately  customer  satisfaction. 


95  Malcolm  Baldrige  was  Secretary  of  Commerce  from  1981  until  his  death  in  a  rodeo  accident  in 
July  1987.  Baldrige  was  a  proponent  of  quality  management  as  a  key  to  this  country’s  prosperity  and 
long-term  strength.  He  took  a  personal  interest  in  the  quality  improvement  act  that  was  eventually 
named  after  him  and  helped  draft  one  of  the  early  versions.  In  recognition  of  his  contributions, 
Congress  named  the  award  in  his  honor. 
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1.5.9  Six  Sigma 

Six  Sigma  provides  organizations  a  tool  for  focusing  on  continuous  improvement 
activities  to  achieve  near  perfection  without  dramatically  increasing  costs.  Sigma  is  a 
term  denoting  one  standard  deviation.  Six  Sigma  will  encompass  99.9997  percent  of  the 
sample.  In  quality  terms,  this  is  approaching  near  perfection.  Three  sigma  or  99.73  per¬ 
cent  perfection,  will  result  in  54,000  incorrect  drug  prescriptions  a  year  or  five  missed 
landings  at  Dulles  International  Airport  each  day.  Six  Sigma  will  only  lead  to  one  incor¬ 
rect  drug  prescription  every  25  years  or  one  missed  landing  in  10  years  at  all  the  U.S. 
airports  combined.  Six  Sigma  process  steps  include  defining,  measuring,  analyzing, 
improving,  and  controlling  processes. 

1.5.10  Technical  Risk  Identification  and  Mitigation  System  (TRIMS) 

TRIMS  is  a  knowledge-based,  process-oriented  technical  risk  identification  and 
management  tool  developed  by  the  Navy’s  Best  Manufacturing  Practices  Center  of 
Excellence  (BMPCOE).  Based  on  the  DoD  4245. 7-M  (Transition  From  Development  to 
Production)  templates  and  NAVSO  P-6071  (Best  Practices:  How  to  Avoid  Surprises  in 
the  World's  Most  Complicated  Technical  Process ),  it  provides  early  and  continuous 
insight  into  technical  risks  and  mitigation  efforts.  TRIMS  is  highly  tailorable,  flexible, 
and  scalable  for  specific  program  needs.  It  is  an  element  of  the  BMPCOE  Program  Man¬ 
ager’s  Work  Station  (PMWS)  toolkit  and  provides  insight  into  technical  process  risks 
before  they  become  downstream  cost  and  schedule  problems. 

1.5.11  Theory  of  Constraints  (TOC) 

TOC  identifies  constraints  in  a  process  to  minimize  their  effect  by  improving 
throughput,  productivity,  and  closely  control  resources  (inventory  and  other  expenses).  A 
constraint  is  a  factor  that  limits  an  organization’s  ability  to  achieve  its  goal.  The  output  of 
a  plant  (or  process)  is  dictated  by  the  bottleneck.  In  TOC  terms,  the  bottleneck  is  called 
the  “drum,”  and  it  paces  the  plant.  “Buffer”  is  the  inventory  in  front  of  the  bottleneck  that 
is  there  to  ensure  that  the  bottleneck  is  never  idle.  The  “rope”  is  the  communication  sys¬ 
tem  used  to  communicate  the  inventory  needs  of  the  bottleneck  back  to  the  material 
release  point.  Control  the  bottleneck  to  control  production.  TOC  process  steps  are 

•  Step  1:  Identify  the  constraint. 

•  Step  2:  Get  more  production  at  that  constraint  with  the  existing  capacity 
limitations. 
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•  Step  3:  Keep  materials  from  sitting  idle  in  a  queue  at  a  nonconstrained 
resource. 

•  Step  4:  Find  other  ways  to  increase  capacity  if  needed  (e.g.,  second  shift). 

•  Step  5:  Go  back  to  step  1. 

1.6  WEB  SITES  RELATED  TO  MANUFACTURING  MANAGEMENT 

Table  1-5  lists  some  Web  sites  related  to  manufacturing  management.  Note:  Some 
sites  may  be  restricted  to  a  .mil  addressee.  Also,  the  government  does  not  endorse  any  of 
the  commercial  sites. 


Table  1-5.  Manufacturing  Management-Related  Web  Sites 


Best  Practices 

Best  Manufacturing  Practices:  http://www.bmpcoe.org 

Industry  Week’s  Census  of  Best  Manufacturing  Practices: 

http://www.industryweek.com/ 

Diminishing 
Manufacturing 
Sources  and 
Material  Short¬ 
ages  (DMSMS) 

Defense  Microelectronics  Activity  (DMEA):  http://www.dmea.osd.mil/ 

DMSMS  GIDEP  Site:  http://www.dmsms.org/  and  http://www.gidep.org/ 

Environmental 

EPA’s  Laws  and  Regulations:  http://www.epa.gov/epahome/lawreg.htm 

The  Defense  Environmental  Network  and  Information  exchange  (DENIX): 

http://sa.kevric.com/eq/eqdenix.htm 

Joint  Group  on  Pollution  Prevention  (JG-PP):  http://www.jgpp.com/ 

Industrial  Base 

DCMA’s  Industrial  Analysis  Center: 

http://www.dcma.mil/communicator/archives/spring%20summer%202003/industrial.htm 
Defense  Manufacturing  in  2010  and  Beyond:  http://books.nap.edu/html/defman/ 

Industrial  Base  Information  Center:  http://www.ml.afrl.af.mil/ibic/ 

Lean 

Lean  Enterprise  Institute:  http://www.lean.org 

Lean  Aerospace  Initiative  (LAI):  http://web.mit.edu/lean/index.html 

ManTech 

DoD:  http://www.dodmantech.com 

Army:  http://www.armymantech.com/ 

Navy:  https://www.navymantech.com/home.html 

Air  Force:  http://www.ml.afrl.af.mil/mlm/ 

Process  Improve¬ 
ment  Tools 

SixSigma:  http://www.isixsigma.com/ 

ASC/EN  Manufacturing  Development  Guide:  http://www.wpafb.af.mil/alpha-index.html 
PMWS  can  be  accessed  from  the  Navy’s  BMPCOE  Web  site  www.bmpcoe.org. 

Statistical  Process  Control  Links  [National  Aeronautics  and  Space  Administration 
(NASA)]:  http://www.hq.nasa.gov/office/hqlibrary/ppm/ppm31  .htm 

Quality 

Baldrige  National  Quality  Program  (BNQP):  http://www.quality.nist.gov/ 

Quality  Digest:  http://www.qualitydigest.com/ 

Quality  Online:  http://qualitymag.com 

Supply  Chain 
Management 

Supply  Chain  Council:  http://www.supply-chain.org/ 

Supply  Chain  Today:  http://supplychaintoday.com/ 
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ACTD 

Advanced  Concept  Technology  Demonstration 

AFRL 

Air  Force  Research  Laboratory 

AKSS 

AT&L  Knowledge  Sharing  System 

AQS 

Advanced  Quality  System 

ASC/EN 

Aerospace  Engineering  Directorate  (WPAFB) 

AT&L 

Acquisition,  Technology,  and  Logistics 

ATD 

Advanced  Technology  Demonstration 

BMPCOE 

Best  Manufacturing  Practices  Center  of  Excellence 

BNQP 

Baldrige  National  Quality  Program 

BOM 

Bill  of  Materials 

CDR 

Critical  Design  Review 

CPD 

Capability  Production  Document 

CPI 

Critical  Program  Information 

CR 

Concept  Refinement 

DAD 

Defense  Acquisition  Deskbook 

DAU 

Defense  Acquisition  University 

DTC 

design-to-cost 

DCMA 

Defense  Contract  Management  Agency 

DENIX 

Defense  Environmental  Network  and  Information 
exchange 

DFARS 

Defense  Federal  Acquisition  Regulations  Supplement 

DFMA 

Design  for  Manufacturing  and  Assembly 

DFX 

Design  for  Excellence 

DLA 

Defense  Logistics  Agency 

DMEA 

Defense  Microelectronics  Activity 

DMSMS 

Diminishing  Manufacturing  Sources  and  Material 
Shortages 

DoD 

Department  of  Defense 
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DoDD 

Department  of  Defense  Directive 

DoDI 

Department  of  Defense  Instruction 

DOE 

Design  of  Experiments 

DRR 

Design  Readiness  Review 

DT&E 

developmental  test  and  evaluation 

DTC 

design-to-cost 

EMRL 

Engineering  and  Manufacturing  Readiness  Level 

EPA 

Environmental  Protection  Agency 

FRP 

Full  Rate  Production 

GIDEP 

Government-Industry  Data  Exchange  Program 

HAZMAT 

hazardous  material 

HMMP 

Hazardous  Material  Management  Program 

HW 

hardware 

IB 

industrial  base 

ICA 

Industrial  Capability  Assessment 

ID 

identify 

IOT&E 

Initial  Operational  Test  and  Evaluation 

IPPD 

Integrated  Product  and  Process  Development 

IPT 

Integrated  Process  Team 

ISO 

International  Organization  for  Standardization 

JDMPT 

Joint  Defense  Manufacturing  Technology  Panel 

JG-PP 

Joint  Group  on  Pollution  Prevention 

KC 

Key  Characteristics 

LAI 

Lean  Aerospace  Initiative 

LCL 

Life-Cycle  Logistics 

LRIP 

Low  Rate  Initial  Production 

M&S 

modeling  and  simulation 

ManTech 

Manufacturing  Technology 

MDA 

Missile  Defense  Agency 

MRL 

Manufacturing  Readiness  Level 

MS 

Milestone 

NASA 

National  Aeronautics  and  Space  Administration 
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NAVSO 

Navy  Staff  Office 

NIST 

National  Institute  of  Standards  and  Technology 

PCA 

Physical  Configuration  Audit 

PEP 

Producibility  Engineering  and  Planning 

PMWS 

Program  Manager’s  Work  Station 

PR 

Program  Element 

PRR 

Production  Readiness  Review 

QFD 

Quality  Function  Deployment 

QM 

quality  management 

R&D 

research  and  development 

RAM 

Reliability,  Availability,  and  Maintainability 

S&T 

science  and  technology 

SBA 

Simulation-Based  Acquisition 

SDD 

System  Development  and  Demonstration 

STE 

Special  Test  Equipment 

SVR 

System  Verification  Review 

T&E 

test  and  evaluation 

TAP 

Technology  Area  Plan 

TD 

Technology  Development 

TOC 

Theory  of  Constraints 

TRIMS 

Technical  Risk  Identification  and  Mitigation  System 

TRL 

Technology  Readiness  Level 

WPAFB 

Wright  Patterson  Air  Force  Base 
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APPENDIX  J. 

EASY-REFERENCE  DISPLAYS  OF 
THE  TRA  ACTIVITIES  TIME  LINE  AND 
THE  HARDWARE/SOFTWARE  TRLs 


J-l 


Suggested  Time  Line  for  TRA  Activities  for  ACAT  ID  and  1AM  Programs 


Task  Name 


m  mi  ms  ms  m  m  m  m  ma  we  ws  mi  we  we  W4  W3  W2  wi  wo  m  m  m  m  m  m  m  mi 


PM  Notifies  CS&T  Exec  and  DUSD(S&T)  of  the  date  for  MS  review  meeting 
DUSD(S&T)  Appoints  Action  Officer  (AO)  and  so  Notifies  PM  and  CS&T  Exec 
PM,  CSSJ  Exec,  and  DUSD(S&T)/A0  Agree  on  TRA  Schedule 
CTE  Identification  Process* 

CTE  TRL  Evaluation  Data  Collection* 

PM  Identifies  CTEs  to  CSSJ  Exec  and  DUSD(SSJ) 

PM  and  CS&T  Exec  Agree  on  CTEs 


CS&T  Exec  Directs  TRA  (Copy  to  AO) 


Component  TRA  is  Performed 

CS&T  Exec  Sends  TRA  to  Component  Acquisition  Exec  (CAE)  and  Copy  to  DUSD(S&T) 
CAE  Accepts  TRA  Findings  or  Reconciles  Them  with  the  PM 


AO  informs  CS&T  Exec  of  adequacy  of  TRA  and  organizes  evaluation  of  TRA 
CAE  Sends  Endorsed  TRA  Findings  to  DUSD(S&T),  with  Notation  of  any  Changes 
AO  Leads  DUSD(S&T)  Evaluation  of  TRA 


AO  Briefs  DUSD(SkT)  on  Evaluation  Status 
(If  necessary,  Independent  Technical  Asessment  (ITA)  Directed  and  Conducted} 
DUSD(S&T)  Sends  Results  of  Evaluation  or  ITA  to  DIPT  and  DAB  or  ITAB 
Milestone  Review  Meeting 


These  actions  can  occur  as  much  as 
3  years  before  the  milestone 


Hardware  and  Software  Technology  Readiness  Levels  (TRLs) 

[Sources:  Defense  Acquisition  Guidebook  (October  2004)  and  IT  TRL  Working  Group  Minutes  (November  2004)] 


Hardware  TRL  Definitions,  Descriptions,  and  Supporting  Information 

Software  TRL  Definitions,  Descriptions,  and  Supporting  Information 

TRL  Definition 

Description 

Supporting  Information 

TRL  Definition 

Description 

Supporting  Information 

1 

Basic  principles 
observed  and 
reported. 

Lowest  level  of  technology  readiness.  Scientific  research  begins  to 
be  translated  into  applied  research  and  development  (R&D).  Exam¬ 
ples  might  include  paper  studies  of  a  technology’s  basic  properties. 

Published  research  that  identifies  the  principles  that  underlie  this 
technology.  References  to  who,  where,  when. 

1 

Basic  principles 
observed  and 
reported. 

Lowest  level  of  software  technology  readiness.  A  new  software 
domain  is  being  investigated  by  the  basic  research  community.  This 
level  extends  to  the  development  of  basic  use,  basic  properties  of 
software  architecture,  mathematical  formulations,  and  general  algo¬ 
rithms. 

Basic  research  activities,  research  articles,  peer-reviewed  white 
papers,  point  papers,  early  lab  model  of  basic  concept  may  be 
useful  for  substantiating  the  TRL  level. 

2 

Technology 
concept  and/or 
application  for¬ 
mulated. 

Invention  begins.  Once  basic  principles  are  observed,  practical 
applications  can  be  invented.  Applications  are  speculative,  and 
there  may  be  no  proof  or  detailed  analysis  to  support  the  assump¬ 
tions.  Examples  are  limited  to  analytic  studies. 

Publications  or  other  references  that  outline  the  application  being 
considered  and  that  provide  analysis  to  support  the  concept. 

2 

Technology 
concept  and/or 
application  for¬ 
mulated. 

Once  basic  principles  are  observed,  practical  applications  can  be 
invented.  Applications  are  speculative,  and  there  may  be  no  proof  or 
detailed  analysis  to  support  the  assumptions.  Examples  are  limited 
to  analytic  studies  using  synthetic  data. 

Algorithms  run  on  a  surrogate  processor  in  a  laboratory  environ¬ 
ment,  instrumented  components  operating  in  laboratory  environ¬ 
ment,  laboratory  results  showing  validation  of  critical  properties. 

3 

Analytical  and 
experimental 
critical  function 
and/or  charac¬ 
teristic  proof  of 
concept. 

Active  R&D  is  initiated.  This  includes  analytical  studies  and  labora¬ 
tory  studies  to  validate  physically  the  analytical  predictions  of  sepa¬ 
rate  elements  of  the  technology.  Examples  include  components  that 
are  not  yet  integrated  or  representative. 

Results  of  laboratory  tests  performed  to  measure  parameters  of 
interest  and  comparison  to  analytical  predictions  for  critical  subsys¬ 
tems.  References  to  who,  where,  and  when  these  tests  and  com¬ 
parisons  were  performed. 

3 

Analytical  and 
experimental 
critical  function 
and/or  charac¬ 
teristic  proof  of 
concept. 

Active  R&D  is  initiated.  The  level  at  which  scientific  feasibility  is 
demonstrated  through  analytical  and  laboratory  studies.  This  level 
extends  to  the  development  of  limited  functionality  environments  to 
validate  critical  properties  and  analytical  predictions  using  noninte- 
grated  software  components  and  partially  representative  data. 

Algorithms  run  on  a  surrogate  processor  in  a  laboratory  environ¬ 
ment,  instrumented  components  operating  in  laboratory  environ¬ 
ment,  laboratory  results  showing  validation  of  critical  properties. 

4 

Component 
and/or  bread¬ 
board  validation 
in  a  laboratory 
environment. 

Basic  technological  components  are  integrated  to  establish  that 
they  will  work  together.  This  is  relatively  “low  fidelity”  compared  with 
the  eventual  system.  Examples  include  integration  of  “ad  hoc” 
hardware  in  the  laboratory. 

System  concepts  that  have  been  considered  and  results  from 
testing  laboratory-scale  breadboard(s).  References  to  who  did  this 
work  and  when.  Provide  an  estimate  of  how  breadboard  hardware 
and  test  results  differ  from  the  expected  system  goals. 

4 

Module  and/or 
subsystem  vali¬ 
dation  in  a  labo¬ 
ratory  environ¬ 
ment  (i.e.,  soft¬ 
ware  prototype 
development 
environment). 

Basic  software  components  are  integrated  to  establish  that  they  will 
work  together.  They  are  relatively  primitive  with  regard  to  efficiency 
and  robustness  compared  with  the  eventual  system.  Architecture 
development  initiated  to  include  interoperability,  reliability,  maintain¬ 
ability,  extensibility,  scalability,  and  security  issues.  Emulation  with 
current/iegacy  elements  as  appropriate.  Prototypes  developed  to 
demonstrate  different  aspects  of  eventual  system. 

Advanced  technology  development,  stand-alone  prototype  solving  a 
synthetic  full-scale  problem,  or  standalone  prototype  processing 
fully  representative  data  sets. 

5 

Component  and/ 
or  breadboard 
validation  in  a 
relevant  environ¬ 
ment. 

Fidelity  of  breadboard  technology  increases  significantly.  The  basic 
technological  components  are  integrated  with  reasonably  realistic 
supporting  elements  so  they  can  be  tested  in  a  simulated  environ¬ 
ment.  Examples  include  “high-fidelity”  laboratory  integration  of  com¬ 
ponents. 

Results  from  testing  a  laboratory  breadboard  system  are  integrated 
with  other  supporting  elements  in  a  simulated  operational  environ¬ 
ment.  How  does  the  “relevant  environment”  differ  from  the  expected 
operational  environment?  How  do  the  test  results  compare  with 
expectations?  What  problems,  if  any,  were  encountered?  Was  the 
breadboard  system  refined  to  more  nearly  match  the  expected 
system  goals? 

5 

Module  and/or 
subsystem  vali¬ 
dation  in  a  rele¬ 
vant  environ¬ 
ment. 

Level  at  which  software  technology  is  ready  to  start  integration  with 
existing  systems.  The  prototype  implementations  conform  to  target 
environment/interfaces.  Experiments  with  realistic  problems. 

Simulated  interfaces  to  existing  systems.  System  software  archi¬ 
tecture  established.  Algorithms  run  on  a  processor(s)  with  charac¬ 
teristics  expected  in  the  operational  environment. 

System  architecture  diagram  around  technology  element  with  criti¬ 
cal  performance  requirements  defined.  Processor  selection  analy¬ 
sis,  Simulation/Stimulation  (Sim/Stim)  Laboratory  buildup  plan. 
Software  placed  under  configuration  management.  COTS/GOTS  in 
the  system  software  architecture  are  identified. 

6 

System/ 
subsystem 
model  or  proto¬ 
type  demonstra¬ 
tion  in  a  relevant 
environment. 

Representative  model  or  prototype  system,  which  is  well  beyond 
that  of  TRL  5,  is  tested  in  a  relevant  environment.  Represents  a 
major  step  up  in  a  technology’s  demonstrated  readiness.  Examples 
include  testing  a  prototype  in  a  high-fidelity  laboratory  environment 
or  in  a  simulated  operational  environment. 

Results  from  laboratory  testing  of  a  prototype  system  that  is  near 
the  desired  configuration  in  terms  of  performance,  weight,  and  vol¬ 
ume.  How  did  the  test  environment  differ  from  the  operational  envi¬ 
ronment?  Who  performed  the  tests?  How  did  the  test  compare  with 
expectations?  What  problems,  if  any,  were  encountered?  What 
are/were  the  plans,  options,  or  actions  to  resolve  problems  before 
moving  to  the  next  level? 

6 

Module  and/or 
subsystem  vali¬ 
dation  in  a  rele¬ 
vant  end-to-end 
environment. 

Level  at  which  the  engineering  feasibility  of  a  software  technology  is 
demonstrated.  This  level  extends  to  laboratory  prototype  imple¬ 
mentations  on  full-scale  realistic  problems  in  which  the  software 
technology  is  partially  integrated  with  existing  hardware/software 
systems. 

Results  from  laboratory  testing  of  a  prototype  package  that  is  near 
the  desired  configuration  in  terms  of  performance,  including  physi¬ 
cal,  logical,  data,  and  security  interfaces.  Comparisons  between 
tested  environment  and  operational  environment  analytically  under¬ 
stood.  Analysis  and  test  measurements  quantifying  contribution  to 
system-wide  requirements  such  as  throughput,  scalability,  and  reli¬ 
ability.  Analysis  of  human-computer  (user  environment)  begun. 

7 

System  proto¬ 
type  demonstra¬ 
tion  in  an  opera¬ 
tional  environ¬ 
ment. 

Prototype  near  or  at  planned  operational  system.  Represents  a 
major  step  up  from  TRL  6  by  requiring  demonstration  of  an  actual 
system  prototype  in  an  operational  environment  (e.g.,  in  an  aircraft, 
in  a  vehicle,  or  in  space).  Examples  include  testing  the  prototype  in 
a  test  bed  aircraft. 

Results  from  testing  a  prototype  system  in  an  operational  environ¬ 
ment.  Who  performed  the  tests?  How  did  the  test  compare  with 
expectations?  What  problems,  if  any,  were  encountered?  What 
are/were  the  plans,  options,  or  actions  to  resolve  problems  before 
moving  to  the  next  level? 

7 

System  proto¬ 
type  demonstra¬ 
tion  in  an  opera¬ 
tional  high-fidel¬ 
ity  environment. 

Level  at  which  the  program  feasibility  of  a  software  technology  is 
demonstrated.  This  level  extends  to  operational  environment  proto¬ 
type  implementations  where  critical  technical  risk  functionality  is 
available  for  demonstration  and  a  test  in  which  the  software  tech¬ 
nology  is  well  integrated  with  operational  hardware/software  sys¬ 
tems. 

Critical  technological  properties  are  measured  against  requirements 
in  a  simulated  operational  environment. 

8 

Actual  system 
completed  and 
qualified  through 
test  and  demon¬ 
stration. 

Technology  has  been  proven  to  work  in  its  final  form  and  under 
expected  conditions.  In  almost  all  cases,  this  TRL  represents  the 
end  of  true  system  development.  Examples  include  developmental 
test  and  evaluation  of  the  system  in  its  intended  weapon  system  to 
determine  if  it  meets  design  specifications. 

Results  of  testing  the  system  in  its  final  configuration  under  the 
expected  range  of  environmental  conditions  in  which  it  will  be 
expected  to  operate.  Assessment  of  whether  it  will  meet  its  opera¬ 
tional  requirements.  What  problems,  if  any,  were  encountered? 

What  are/were  the  plans,  options,  or  actions  to  resolve  problems 
before  finalizing  the  design? 

8 

Actual  system 
completed  and 
mission  qualified 
through  test  and 
demonstration  in 
an  operational 
environment. 

Level  at  which  a  software  technology  is  fully  integrated  with  opera¬ 
tional  hardware  and  software  systems.  Software  development 
documentation  is  complete.  All  functionality  tested  in  simulated  and 
operational  scenarios. 

Published  documentation  and  product  technology  refresh  buiid 
schedule.  Software  resource  reserve  measured  and  tracked. 

9 

Actual  system 
proven  through 
successful  mis¬ 
sion  operations. 

Actual  application  of  the  technology  in  its  final  form  and  under  mis¬ 
sion  conditions,  such  as  those  encountered  in  operational  test  and 
evaluation  (OT&E).  Examples  include  using  the  system  under 
operational  mission  conditions. 

OT&E  reports. 

9 

Actual  system 
proven  through 
successful  mis¬ 
sion-proven 
operational 
capabilities. 

Level  at  which  a  software  technology  is  readily  repeatable  and 
reusable.  The  software  based  on  the  technology  is  fully  integrated 
with  operational  hardware/software  systems.  All  software  docu¬ 
mentation  verified.  Successful  operational  experience.  Sustaining 
software  engineering  support  in  place.  Actual  system. 

Production  configuration  management  reports.  Technology  inte¬ 
grated  into  a  reuse  “wizard”;  out-year  funding  established  for  sup¬ 
port  activity 

